Transrectal Prostate Ultrasonography

Vol. 158. 856-860. September 1997 Printed in U.S.A.

TRANSRECTAL PROSTATE ULTRASONOGWHY: VARIABILITY OF INTERPRETATION MARTIN I. RESNICK,* JOSEPH A. SMITH, JR., PETER T. SCARDINO, MARLENE J. EGGER, ALBERT0 HERNANDEZ AND STEVEN C . ROSE From the Departments of Urolog.v, Case Western Reserue University, Cleveland, Ohio, Vanderbilt University, Nashville, Tennessee, Baylor College of Medicine, Houston, Texas, and Divisions of Urology, Radiology, and Family and Preventive Medicine, University of Utah, Salt Lake City, Utah

ABSTRACT

Purpose: The current study was designed to compare the interpretation of the individual performing transrectal ultrasound examination (operator) with experienced individuals who interpreted the examination with and without the availability of clinical data. Inter-observer and intra-observer variability was compared to determine the reproducibility and reliability of the study. Materials and Methods: All patients undergoing radical prostatectomy for treatment of localized carcinoma of the prostate underwent a transrectal ultrasound examination before the procedure. The sonogram was interpreted by the operator and reviewers. The radical prostatectomy specimen was examined pathologically and the staging as determined by ultrasound was compared with the pathological findings. Results: Ultrasound operator accuracy for extracapsular extension and seminal vesicle invasion was 0.70 and 0.74, respectively, compared with the accuracy of the reviewers, which ranged from 0.59 to 0.75 and 0.44 to 0.74 for extracapsular extension and seminal vesicle invasion, respectively. In general, blinded reviews were less accurate than unblinded reviews but this was only statistically significant for 2 reviewers. Conclusions: Although for most reviewers the addition of clinical data did not improve the accuracy of the interpretation, a n advantage was noted for the operator, that is, the individual performing the examination. In general, the technical quality of the examination was related to the accuracy of the readings. KEY WORDS:prostatic neoplasms, ultrasonography Transrectal ultrasonography has been used in a variety of applications in the assessment of patients with disorders of the prostate. The procedure has been used to assist in biopsy, assess for nonpalpable intraprostatic abnormalities in patients with elevated prostate specific antigen (PSA), stage cases with an established diagnosis of carcinoma of the prostate, and monitor response to therapy, including surgery, radiation therapy and endocrine therapy. Experience indicates that many carcinomas can be visualized as peripheral zone hypoechoic areas but it is well recognized that many carcinomas are isoechoic and cannot be visualized, and not all peripheral zone hypoechoic areas represent malignancy. In a similar manner strict criteria have not been established for distinguishing between confined and extracapsular disease, and studies have varied as to the efficacy of this modality in determining tumor extent.’.’ Because no strict and reproducible criteria have been established for the diagnosis and staging of carcinoma of the prostate it has been noted that significant variability may exist in the interpretation of transrectal prostate ultrasound images, but, to our knowledge, no prospective studies have addressed this issue. Additionally, it is well recognized that the accuracy of diagnostic studies vary and an assessment of transrectal ultrasonography has not been done.”-fi The cur-

rent study was designed to compare the interpretation of the individual performing the ultrasound examination (operator) with experienced individuals who interpreted the examination with and without the availability of clinical data. Observer variability was compared to determine the reproducibility and reliability of the study. The findings of the ultrasound examination were compared to the pathological findings following radical prostatectomy. MATERIALS AND METHODS

All patients were evaluated by complete history and physical examination. All reviewers (5 urologists and 1 radiologist) were considered experienced in the performance and interpretation of transrectal ultrasound. Each had incorporated transrectal ultrasound into clinical practice for a t least a year before initiation of this study and had acknowledged regional if not national expertise in interpreting ultrasound imaging of the prostate. A series of training sessions among the investigators was held, which during the criteria for the performance of transrectal ultrasound and interpretation of the results were established. Two experienced physicians performed a digital rectal examination on all patients and independently recorded their findings. A history of diseases that might affect the ultrasound findings was obtained, including chronic prostatitis, epididymitis, prostatodynia and Accepted for publication January 17, 1997. Supported by National Institutes of Health Grants R 0 1 chronic urinary tract infection. Inclusion criterion was the CA47264-03and R01 CA47254-03, and the Utah Regional Cancer presence of localized carcinoma of the prostate considered Center Grant CA 42014. *Requests for reprints: Dr. Martin I. Resnick, De artment of resectable by radical prostatectomy. Transrectal ultrasonogCase Western Reserve University, University Rospitals of raphy was performed within 6 weeks of radical prostatecgc::f%d, 11100 Euclid Ave., Cleveland, Ohio 44106-5046. tomy and a minimum of 2 weeks after prostatic biopsy. EX856

INTERPRETATION OF TRANSRECTAL PROSTATE ULTRASONOGRAPHY

857

elusion criteria were any form of radiotherapy to the prostate label, and fixed in cold formalin for 48 hours. Sections were or hormonal manipulation, performance of a transurethral then embedded in paraffin, cut a t 6 p. and stained with prostatic resection within 4 weeks of the transrectal ultra- hematoxylin and eosin. Slides were traced on paper in sesound and an inability t o perform ultrasound examination quential order, and the cancer and dysplasia were mapped. Additional pathological information was collected by descripbecause of the presence of an anorectal abnormality. A1 patients underwent transrectal ultrasonography at 1 of tion of each region of the prostate as given for the ultrasound 4 institutions. The same machine (Bruel & Kjaer 1856 with a with respect to the presence or absence of tumor, capsular 7 MHz. transducer) was used for all examinations. The pa- invasion, positive surgical margins and number of discrete tient was placed in the left lateral decubitus position and tumors. The volume of the prostate transition zone and tutransrectal ultrasound was performed. The bi-probe was mor was calculated. Invasion of the seminal vesicles, bladder placed in the rectum and 5 mm. scans were obtained from the neck, urethra, external sphincter and extracapsular extenapex to the base in the axial plane. Sagittal scans were sion was noted. Each study site had a designated pathologist. The patholperformed and images were obtained to provide 5 oblique views, including 2 from the left side, 2 from the right side and ogists used a common technique for whole mount sectioning 1 in the midline. Anteroposterior, transverse and longitudi- of the prostate and for reporting results on the forms develnal measurements were obtained on all studies. The presence oped for the study. They were involved in the process of of hyperechoic and hypoechoic areas and their location developing the forms and discussed the methods for preparation of specimens to ensure uniformity in the mode of within the prostate were noted. The sonogram was interpreted by the operator and re- interpretation. The data were input a t the clinics and sent on disk to the corded on a n x-ray camera printout. Reviewers of the initial and repeat reviews evaluated the images with and without coordinating center. Receiver operating characteristics the availability of clinical information (physical findings, se- (ROC) curves were calculated by maximum likelihood estirum PSA, acid phosphatase, biopsy). Data used in the assess- mation under the binormal assumption and tested by the ment of the prostate images included the presence of calcifi- generalized likelihood ratio test when possible using the procation, hypoechoic areas, asymmetry of the margin of the grams of Metz.7 When maximum likelihood estimates failed prostate and abnormalities of the seminal vesicles. An as- to converge, estimates of the area under the ROC curve were sessment was made as to the probability (grade 1 to 5 ) of the obtained by the trapezoidal rule (Wilcoxon estimate), as presence or absence of malignancy and extracapsular disease given by Hanley and McNeil." An analysis of the variability of across reviewers, within rereviews by the same reviewer, (1-definitely absent, 2-probably absent, 3-possibly and between the standard film review and the review includpresent, 4-probably present, 5-definitely present). The reviews were performed by first submitting all rele- ing focus films and clinical data was performed by the nonvant transrectal ultrasound images to the statistical center, parametric method of DeLong et al.9 This analysis was supand they were separated into 2 separate packets for each plemented by a mixed model repeated measures analysis of patient. The first packet was marked "films only," and in- variance on the ROC curve areas based on the cases reviewed cluded the axial and sagittal images of the prostate. Al- and rereviewed. The study was approved by institutional though all patients in this study had known carcinoma of the review boards a t all institutions and all patients consented to prostate, no additional information was provided before in- participate in the study. Details of the characteristics of the terpretation. The second packet was labeled "films plus clin- patient population and methodology have been described in a ical," and included additional spot films of areas of interest related report.'" on the examination. For example, there may have been an enlarged and detailed view of the right base of the prostate if RESULTS that was a n area of induration on digital rectal examination. Variations in reading between individuals and variations The standard digital rectal examination form developed by the group was also included in the packet. This form in reading which 1 individual rereviews the same material. showed the location of any palpable areas of abnormality Table 1represents the initial review only and table 2 displays along with a key indicating the level of suspicion of the the accuracy at initial and rereview for 3 reviewers who rated examiner for each palpably abnormal area. The reviewer was the same 80 patients. Tables 3 and 4 compare operator ROC also provided with the serum PSA level. The reviewer then area versus reviewer ROC area for the films only, and films completed the same set of forms given this detailed informa- and clinical data review for extracapsular extension and seminal vesicle invasion. tion. As measured by the area under the ROC curve, initial To determine intra-observer variability, a random sampling of films was also selected for a repeat review. Packets of reviewer accuracy in the review of standard films ranged films prepared in the same manner as for the initial review from 0.59 to 0.75 for extracapsular extension and 0.44to 0.74 were then mailed again to the reviewers. The same process for seminal vesicle invasion. The accuracy of the reviews of was followed with this reinterpretation wherein forms were standard films supplemented with focus films and clinical completed using standard films only and then with the avail- data ranged from 0.44 to 0.74 for extracapsular extension ability of clinical information. In most cases a period of sev- and 0.39 to 0.81 for seminal vesicle invasion. This is in eral months had elapsed between the initial and repeat re- comparison to ultrasound operator accuracy of approxiviews. The reviewers were blinded as to the results of their mately 0.70 for extracapsular extension and 0.74 for seminal vesicle invasion. Table 5 displays the average area under the initial assessment. All patients underwent radical prostatectomy and the ROC curve for 6 reviewer initial reviews (table 1).It appears specimen was prepared for whole mount sectioning. All pros- that the average reviewer could be considered to have 64 to tate specimens were fixed for a t least 2 days in cold formalin. 687c accuracy in the evaluation of extracapsular extension Following fixation the apex was removed and cut in a manner and just more than 60% accuracy in the determination of of a cervical cone for evaluation of the apical margin. The seminal vesicle invasion with little information added by remainder of the cuts were anterior to posterior and 3 to 5 focus films and clinical data. Tables 6 and 7 display the accuracy for extracapsular exmm. thick, and extended from near the apex to the seminal vesicles. Each cut was photographed, surgical margins inked, tension and seminal vesicle invasion, respectively, by the test right side was nicked with a scalpel and the side to be cut was of DeLong et al.9 For extracapsular extension reviewers 1 first marked with merbromin. Each enlarged specimen was and 3 had an advantage in the initial review of having seen placed in a n embedding bag with excision number and sub- clinical data during film review. Initial review of reviewer 3

858

INTERPRETATION OF TRANSRECTAL PROSTATE ULTRASONOGRAPHY TABLE1. Areas under ROC-curire for initial reuiew

~~

Films Reviewer

Extracapsular Extension

Seminal Vesicle Invasion

I 0 13

0 OH 0.05

0.39 2 0.07 0.74 t 0.04

0.05

0 69

I 0 07

6 LJltrasound operator

0.44 0.50

?

Digital rectal c’xamination by Ultrasound operator

0.67

t 0.48 2 0 05

z 0 08

0.78

*

?

0.72

0.05

f

f

No 123 143 174 88 110 Extracapsular extension 107 Seminal vesicle invasion 135 257 Extracapsular extension 256 Seminal vesicle invasion 257 Extracapsular extension

0.05

____~ ~~~

Review of Standard Films Only

Clinical Data Review

0.59 t 0.07 0.62 I 0.07 0.67 f 0.05 0.71 I 0.07 0.68 2 0.08

0 69 0.73 0 84 0 74 0 65

Diptal rectal examination hy examiner 2

T

-

Extracapsular Extension

Seminal Vesicle Invasion

0.63 0.59 0.62 0.i5 0.66

f 0.07 t 0.06 z 0.07 t 0.06 z 0.11

0.61 1 0.08 0.62 t 0.10 0.60 2 0.05 0.74 z 0.08 0.67 1 0.08

120 143 174 88 110 Extracapsular extension 108 Seminal vc.sirlr in-

0.59 f 0.08

0.44 I O . 1 0

135

No.

~.d b:i m

256 Seminal vesicle invasion 251 Extracapsular extension

0.04

249 Seminal vesicle invasion

-

TABLE2 . Areas under ROC curue for initial rereciews by 3 reviewers (80patients)” ~

Films

t

Review of Standard Films Only

Clinical Data Review -

Extracapsular Extension Initial review: 1 2

3 Ultrasound operator Rereview: 1

0.71 0.73 0.83 0.70

..~ ~~

Seminal Vesicle Invasion

Extracapsular Extension

Seminal Vesicle Invasion

0.63 2 0.10 0.57 t 0.14 0.78 t 0.08 0.68 ? 0.08

0.59 z 0.08 0.52 t 0.09 0.56 t 0.10

0.61 2 0.11 0.57 ? 0.16 0.67 t 0.09

f 0.08

t 0.06 t 0.07 t 0.07

0.78 t 0.07 0.58 t 0.08 0.80 ‘t 0.07 0.79 ? 0.07 0.76 t O.OR 0.61 t 0.10 0.85 % 0.05 2 0.67 3 0.09 3 0.56 t 0.07: 0.71 t 0.09 0.55 t 0.08 0.62 3 0.10 * For the rereview. the reviewers were unaware of the results of their initial review which had been conducted at a different time. This rereview was performed to determine intra-observer variability. + Areas computed nonparametrically via Hanley and McNeil method.* ~

TABLE3. Operator ROC area versus reviewer ROC area Films

Films Only Review Reviewer

No. Reviewer Area

Operator Area

p Value*

Reviewer Area

+ Clinical Data

Review

Operator Area

p Value’

0.70 0.73 0.83 0.73 0.74 0.44

0.70 0.70 0.70 0.73 0.63 0.70

0.8391 0.2791 0 2719 0.9777 0.6752 0.0055

0.3980 0.3219

0.69 0.68

0.70 0 70

0.7395 0 6766

-

-

Extracapsular extension for initial review 1 2 3 4 5 6

12+6R 12+68 12+68 11467 5+28 19+115

0.59 0.53 0.56 0.74 0.73 0.58

0.70 0.70 0.70 0.73 0.64 0.70

0.5835 0.2152 0.3485 0.7942 0.7971 0.3912

Extracapsular extension for rereview

*

1 80 0.58 0.70 2 80 0.61 0.70 80 3 From the generalized likelihood ratio test for paired data of Metz.’

was significantly more accurate than his rereview. For seminal vesicle invasion this method revealed that reviewer 3 had significantly better accuracy than reviewer 2 on the initial review of films plus clinical data and initial films only review of reviewer 2 was less accurate than his rereview. Comparison of blinded and unblinded reviews. Tables 6 and 7 also address blinded (films only) and unblinded (films and clinical data) reviews again by the test of DeLong e t a1.Y In general, the blinded reviews were less accurate than unblinded reviews but this was only statistically significant for reviewers 1 and 3, and only in their initial reviews for extracapsular extension.

-

-

Comparison of operator with standard review (blinded and unblindedl By the test of DeLong et a1 (tables 6 and 7 ) ultrasound operator accuracy was sometimes better and sometimes worse than individual reviewer accuracy for the 80 patients reviewed and rereviewed by all 3 reviewers. There was no statistically significant advantage of ultrasound operator review over the review of either blinded or unblinded static films. No statistically significant advantage of the operator was found by the test of DeLong et a1 when compared with the 3 reviewers who performed rereviews (reviewers who did not perform rereviews were excluded). We also compared reviewer accuracy to that of t h e ultrasound

859

INTERPRETATION OF TRANSRECTAL PROSTATE ULTRASONOGRAPHY

-

~~~

~

~~~

~

TABLE4. Operator ROC __

~~~~~~~~~

U F ~ S U Sreviewer

ROC area ~~~

~

Films

Films Only Review Reviewer

i

Clinical Data Review

~-

No.

Reviewer Area

Operator Area

p Value*

Reviewer Area

Operator Area

p Value*

0.63 0.56 0.78 0.72 0.71 0.40

0.68 0.68 0.68 0.77 0.42 0.70

0.8204 0.6594 0.4007 0.4461 0.1354 0.0091

0.79 0.76 0.71

0.68 0.68 0.68

0.3876 0.5616 0.8417

Seminal vesicle invasion for initial review 1 2 3 4 5 6

677 13

0.62

67t13 67-tl3 60+17

0.57 0.67 0.74 0.81 0.44

24+8

103+30

0.68 0.68 0.68 0.77 0.50 0.70

0.7594 0.7176 0.9386 0.6974 0.1380 0.0310

Seminal vesicle invasion for rereview 80 0.78 2 80 0.82 3 80 0.62 From the generalized likelihood ratio test for paired data 1

*

0.68 0.68 0.68 of Metz.’

TABLE 5 . Average ROC areas for initial review* Films

2

Films Only Review

Clinical Data Review

0.64 2 0.06 0.61 Z 0.10 * Six ROC curves for each cell. There was surprisingly little change in the area under the curve aRer the reviewer had clinical information available. 0.68 2 0.14

0.5292 0.3881 0.3778

TABLE 7. Results of DeLong et a1 method comparing accuracy for seminal vesicle invasion of ultrasound operator and 3 reviewers under all conditions Area Under ROC Curve (trapezoidal estimate)*

0.61 i 0.12

TABLE6 . Results of DeLong et a19 method comparing accuracy for extracapsular extension of ultrasound operator and 3 reviewers under all conditions Area Under ROC Curve (Trapezoidal estimate)* Films + Clinical Data

Films Only

Reviewer 1: Initial review 0.69t 0.60i 0.66 0.57 Rereview Reviewer 2: 0.69 0.54 Initial review 0.65 0.59 Rereview Reviewer 3: Initial review 0.73iJ 0.54t Rereview 0.561 0.55 Ultrasound operator 0.68 A total of 80 patients was reviewed under each condition (n’ 68 and n 1 3 , p