Full opinion text
NEWMAN, Circuit Judge: This employment discrimination suit pursuant to Title VII of the Civil Rights Act of 1964, 42 U.S.C. § 2000e-2, once again requires this Court to venture into the complex realm of testing and test validation. The test at issue was designed by New York City officials and administrated on June 30, 1979 to 36,797 applicants for positions on the City’s police force. Plaintiffs are the Guardians Association of the New York City Police Department, Inc., an organization of Black police officers, the Hispanic Society of the New York City Police Department, Inc., an organization of Hispanic police officers, and eight individual Black or Hispanic applicants. Defendants are the New York City Department of Personnel, which performed much of the test preparation, the New York City Civil Service Commission, and the New York City Police Department. The United States District Court for the Southern District of New York (Robert L. Carter, Judge) found that use of the test unjustifiably discriminates against Blacks and Hispanics in violation of Title VII. Guardians Ass’n v. Civil Service Commission, 484 F.Supp. 785 (S.D. N.Y.1980). The Court ordered a broad remedy, including a 50% minority hiring quota. We affirm the District Court’s finding that the City’s specific use of the test violates Title VII, but vacate the remedy and remand for entry of a revised decree. I. Factual Background The test in question, designated Exam No. 8155, was designed to select candidates for hiring as entry-level police officers. Those who pass the exam are selected, in rank order of their test scores, to complete the other aspects of the hiring process-a medical examination, a physical agility test, a psychological test, and a character investigation. These last four components of the hiring process are scored only on a pass/fail basis. Thus, an appellant’s score on Exam No. 8155 is a major determinant of his prospects for becoming a police officer. It is also the only feature of the process alleged to have a discriminatory impact. Once an applicant scores high enough to be selected for the final four hiring steps and successfully completes those steps, he or she becomes a sworn police officer and enters the police academy for five months of training. While successful completion of the training program is a requirement of continuing as a police officer, the Department does not use the training program as a selection device, but anticipates that nearly all academy entrants will go on to active duty. The exam was developed by a fairly elaborate two-stage process; the first stage was an analysis of the police officer’s job, and the second was construction of the test itself. The job analysis consisted of five separate steps. First, the Department of Personnel identified 71 tasks that police officers generally perform, based on interviews with 49 police officers and 49 supervisors. Second, a panel of seven officers and supervisors reviewed the list to add any tasks that had been omitted, and to eliminate those items that were duplicative, or too specialized to be performed by entry-level officers. The result was a consolidated list of 42 entry-level tasks. Third, a questionnaire was distributed to 5,600 police officers, requesting them to rate each of the 42 tasks on the basis of its frequency of occurrence, its importance, and the amount of time normally spent in performing it. The 2,600 responses that were received were then analyzed by computer to yield a ranking of the 42 tasks, according to the combined rating of all the responses. In addition, faculty members of John Jay College were asked to observe police officers during an entire tour of duty and record the tasks that they performed; their survey generally confirmed the identification of the 42 tasks. In the fourth step of the job analysis, the Department of Personnel divided the list of 42 ranked tasks into clusters of related activities. Five such clusters were established: the arrest process, providing assistance to people, police operations, station-house activities, and handling unusual and other occurrences. The fifth step was an analysis of all five clusters, each one by a separate panel of police officers, to identify the “knowledge, skills and abilities” required to perform these tasks at the entry level, and to assign percentages reflecting the relative importance of each of the identified knowledges, skills, and abilities for the cluster as a whole. One panel listed five such qualities for its cluster, all of which are properly characterized as “abilities” or “skills” (hereafter referred to as “abilities”): recalling facts, filling out forms, understanding and applying statutory definitions of crimes, understanding written instructions and applying appropriate procedures, and human relations skills, including communication techniques. Each of the other four panels used the first panel’s list of abilities, but developed its own percentages to express the relative importance of each ability to the tasks within its cluster. The second major stage in developing Exam No. 8155, the process of test construction, consisted of four identifiable steps. First, the percentages of the five abilities necessary to perform each of the five task clusters were multiplied by the weightings that had been given to each task in Step 3 of the job analysis on the basis of frequency, importance, and time spent. This yielded a general measurement for the importance of each of the five abilities for performance of the job of police officer. As a result of this computation, the Department of Personnel concluded that on a test with 100 questions, 15 questions should test for the ability to recall facts, 9 questions for filling out forms, 14 questions for understanding and applying sections of the criminal law, 32 questions for understanding written instructions and applying appropriate procedures, and 30 questions for human relations skills. Next, a group of eleven police officers was selected to write multiple-choice questions that tested for the five abilities, as they related to the 42 identified tasks. The officers wrote many of these questions from Police Academy materials and similar sources, however, without having access to descriptions of the five identified abilities, or the 42 ranked tasks. In the third step, Department of Personnel staff members who did have access to the description of abilities and the ranking of tasks reviewed the questions written by the police officers to assure that the questions were not ambiguous, overly complex, overly specialized, or dependent on prior knowledge. As a result of this review, some questions were discarded, others were revised, and still others were added. Finally, the resulting questions were subjected to a further review by a panel of six police experts,- and by various members of the Department of Personnel. The test that resulted consisted of 100 multiple-choice questions, designed so that the candidate could answer correctly without knowledge of any information beyond what was provided on the test itself. The test materials were determined by the Department of Personnel to require an eighth-grade reading level, on the average, although the 14 questions on law required college-level reading ability. The estimated time for completing the exam was IV2 hours, but 3V2 hours were allowed. The first part of the exam, designed to measure the ability to recall facts, consisted of a page-and-a-half description of a burglary, and a series of 15 questions to be answered without referring back to the description. In the second part, testing ability to fill out forms, the candidates were given a simplified arrest form, and a page-long description of both a robbery and an arrested suspect, and then asked 9 questions about the proper entries to be made in filling out the form. Part three, intended to test ability to apply provisions of law, consisted of 14 questions, each briefly presenting the facts of an incident, and then requiring the candidate to identify the precise criminal offense involved on the basis of definitions provided in the test materials. The remaining 62 questions, of which 32 were intended to measure the ability to follow appropriate procedures and 30 were intended to measure human relations skills, consisted of general instructions as to procedures or appropriate responses for certain types of situations, a description of a specific situation, and then one or more questions asking the proper response to the situation presented. Three of the questions dealing with appropriate procedures, for example, involved the proper response to a bomb threat. Four of the questions in the human relations section involved the proper way to deal with a person who appears to be mentally ill. The test was scored from zero to one hundred, with one point given for each correct answer, and bonus points given for veterans. The candidates were then rank-ordered on the basis of their scores. Scores were generally high, with 13% of the applicants scoring 98 or above, and fully 50% scoring 91 or above. Because of the number of candidates taking the test and the bunching of candidates in the upper range of scores, each point a candidate achieved made a substantial difference in his position on the rank-ordering list. More than 2,000 applicants achieved a score at each numerical grade from 92 to 97. The passing grade was determined in the following manner. The Police Department first estimated that 4,000 police officers would be hired during the four-year period for which the eligibility list resulting from Exam No. 8155 would be valid. The Department further estimated that only one out of three applicants who passed Exam No. 8155 would successfully complete all the remaining steps in the hiring process. Therefore, if this eligibility list was to meet the Department’s needs, 12,000 persons had to pass the exam to provide the 4,000 needed police officers. With all this in mind, the Department simply set the passing grade at the score achieved by the 12,000th highest scoring candidate, which turned out to be 94. Because of the bunching phenomenon, a large number of candidates, 2,124, achieved this same score, so that the actual number who received a passing grade was 13,749. Of the 36,797 applicants who took the test, 6,142 identified themselves as Black, 5,239 identified themselves as Hispanic, 19,-798 identified themselves as White, and 4,847, or 13.2% did not identify their race. Thus, identified Blacks constituted 16.7% of the total applicants, and 19.7% of all those who identified their race, while the equivalent percentages for identified Hispanics were 14.2% and 16.8%, and for identified Whites, 53.8% and 64.5%. Of those who passed the exam, i. e., scored 94 or better, 7.6% had identified themselves as Black and 7.8% had identified themselves as Hispanic, for a known minority population in the passing group of 15.4%, against a known minority population in the applicant pool of 30.9%. In contrast, 66.6% of the passing applicants had identified themselves as White, although Whites comprised only 53.8% of the applicant pool. Viewed in another and more revealing way, the figures show that, among those who had identified themselves by race, the passing rate for Whites was 45.9% compared to 17% for Blacks and 20.5% for Hispanics. The combined minority pass rate was thus about two-fifths of the pass rate for Whites. The Police Department accepted 415 candidates from the list in November, 1979 and planned to hire another 380 in January, 1980. Of this group of 795 candidates, 89.2% were White, 3.5% were Black, and 6.8% were Hispanic. The selection rates (number chosen compared to number of applicants) for these first two uses of the list were 0.5% for Blacks, 1% for Hispanics, and 3.6% for Whites. II. The District Court’s Prior Proceedings and Decision The plaintiffs filed their complaint in this suit, together with a motion for a preliminary injunction, in October, 1979, before any candidates had been accepted on the basis of the list. They charged that the intended use of the lists by the Police Department constituted discrimination against Blacks and Hispanics in violation of the Fourteenth Amendment, Title VII, and various other Federal and state laws. The District Court, by consent of the parties, consolidated the hearing on the preliminary injunction and the trial on the merits. This proceeding was held on November 13, 14, and 15, 1979, shortly after the Police Department’s first use of the list. On January 11, 1980, three days before the Department intended to use the list to accept a second group of trainees, the District Court held a second hearing. That same day, the Court issued an opinion, which was subsequently re-issued in revised form on January 23. The Court’s basic conclusion was that Exam No. 8155 violated Title VII. In reaching this conclusion, the Court used the common mode of Title VII analysis, in which the plaintiff is first required to establish a prima facie case on the basis of disparate impact, and then the defendant is required to rebut the plaintiff’s case by proving that the disparity results from legitimate, job-related selection procedures. The Court first found that the disparity between the percentage of minority group members who achieved a passing score and the percentage of minority group members in the applicant pool was sufficient to establish a prima facie case. It based this finding of disparate impact on the standards developed by the Supreme Court in Castaneda v. Partida, 430 U.S. 482, 97 S.Ct. 1272, 51 L.Ed.2d 498 (1977), and by the Equal Employment Opportunity Commission (EEOC) in its Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. § 1607 (1979). Castaneda stated that, in cases involving large samples, “if the difference between the expected value [from a random selection] and the observed number is greater than two or three standard deviations,” a prima facie case is established. 430 U.S. at 496 n.17, 97 S.Ct. at 1281 n.17. The Uniform Guidelines provide that “[a] selection rate for any race, sex, or ethnic group which is less than four-fifths (Vs) (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact.” § 1607.4(D) [hereinafter Guideline sections are cited only by the subdivisions of 29 C.F.R. § 1607]. The District Court then noted that the discrepancy between the percentage of minority group members in the applicant pool and the percentage of minority group members who passed the test was 39 standard deviations. The evidence also showed that the passing rate of the minority group members was 44.3% of the passing rate of Whites, or about two-fifths. Having concluded that the plaintiffs had established a prima facie case, the Court next concluded that the test was not sufficiently valid to constitute a legitimate attempt to choose those applicants who would become better police officers. Relying primarily on the EEOC Guidelines, the Court stated that a determination of validity based on the content of the test would be inappropriate, first because the test purported to measure abilities that the accepted applicants would be trained to acquire, see Guidelines, § 14(C)(1), and second, because the test actually measured constructs, not abilities, see id. Moreover, the Court concluded that the job analysis was not sufficiently precise to satisfy the Guideline requirement even for content validation, see Guidelines § 14(C)(2). Since content validation was the only method of validation which the City attempted, the Court concluded that the test was invalid, and thus an inadequate rebuttal to the plaintiffs’ prima facie case. In fashioning relief, the District Court noted that a previous examination administered by the New York City Police Department had been found to be in violation of Title VII. Guardians Association v. Civil Service Commission, 431 F.Supp. 526 (S.D.N.Y.), vacated and remanded on other grounds, 562 F.2d 38 (2d Cir. 1977). Concluding that the defendants had “persisted in devising and utilizing testing procedures that continue to discriminate against blacks and hispanics,” the District Court found that the defendants’ “studied adherence to discriminatory procedures must at this point be deemed conscious and deliberate.” 484 F.Supp. at 798. On this basis, the Court held that “[ajffirmative action is mandated as an interim measure either until such discrimination has been totally eliminated or until defendants proceed to select police officers under procedures that are in full compliance with Title VII.” Id. at 799. It enjoined the City from using Exam No. 8155, although permitting it to use the eligibility list from that exam for purposes designated by the Court. In its order, also issued on January 11, the Court ordered the Police Department to achieve at least 30% minority composition of the force, a level comparable to the percentage of minorities in the labor force of the relevant hiring area. To achieve this goal, the Court further ordered that the defendants should “as an interim goal appoint 50% of their entry level police officers from among qualified black and hispanic applicants.” Finally, the Court awarded the plaintiffs attorneys’ fees and costs, and retained jurisdiction “for such further relief or other orders as may be necessary or appropriate to enforce and insure rights to equal employment opportunity within the New York City Police Department.” The City moved to stay the District Court’s order pending consideration of its appeal from the decision. This Court denied the motion. However, we granted a conditional stay, in view of the City’s declared need to hire new police officers and set an expedited schedule for the appeal. III. The Framework of Title VII Analysis As the District Court concluded, the accepted procedure for Title VII cases is to require the plaintiffs to. establish a prima facie case, and then to require the defendants to rebut this showing with proof that the test was legitimately job-related. See Albemarle Paper Co. v. Moody, 422 U.S. 405, 95 S.Ct. 2362, 45 L.Ed.2d 280 (1975); McDonnell Douglas Corp. v. Green, 411 U.S. 792, 93 S.Ct. 1817, 36 L.Ed.2d 668 (1973); Griggs v. Duke Power Co., 401 U.S. 424, 91 5. Ct. 849, 28 L.Ed.2d 158 (1971). The Court correctly concluded that a prima facie case had been established. By any reasonable measure, including the standard deviation rule of Castaneda, supra, or the four-fifths rule of the EEOC Guidelines, Exam No. 8155 had a disparate racial impact. The City argues that statistics alone, specifically a comparison of the racial composition of the passing group to that of the applicant group, are not sufficient to establish a prima facie case. But statistics showing a significantly disparate racial impact have consistently been held to create a presumption of Title VII discrimination. See International Brotherhood of Teamsters v. United States, 431 U.S. 324, 339, 97 S.Ct. 1843, 1856, 52 L.Ed.2d 396 (1977); United States v. City of Chicago, 549 F.2d 415, 428 (7th Cir.), cert. denied, 434 U.S. 875, 98 S.Ct. 225, 54 L.Ed.2d 155 (1977); Kirkland v. New York State Department of Corrections, 520 F.2d 420, 425 (2d Cir. 1975), cert. denied, 429 U.S. 823, 97 S.Ct. 73, 50 L.Ed.2d 84 (1976); Vulcan Society v. Civil Service Commission, 490 F.2d 387, 392-93 (2d Cir. 1973); Chance v. Board of Examiners, 458 F.2d 1167, 1172-73 (2d Cir. 1972). The City also claims that finding a prima facie Title VII violation by state or local governments without a showing of discriminatory intent violates the Tenth Amendment. This view has been definitively rejected by the Seventh Circuit in United States v. City of Chicago, 573 F.2d 416, 422-24 (7th Cir. 1978), and we agree with that analysis. Congress may enforce the Fourteenth Amendment by legislation that prohibits practices the Amendment might not of its own force condemn. See Katzenbach v. Morgan, 384 U.S. 641, 86 S.Ct. 1717, 16 L.Ed.2d 828 (1966). The real issue in this case, therefore, is whether the defendants have rebutted the plaintiffs’ prima facie case by proving that its test was job-related: that the test accurately selected applicants who would be better police officers. Adjudication of this issue presents a more complex problem in the present case than it has in many previous Title VII suits. Many of the previous suits involved tests that were so artlessly constructed that they could be judged invalid without extensive inquiry, fine distinctions, or a precise notion of where the line between validity and invalidity was located. See, e. g., Griggs, supra, 401 U.S. at 431, 91 S.Ct. at 853 (intelligence tests used “on the Company’s judgment that they generally would improve the overall quality of the work force”); United States v. N. L. Industries, Inc., 479 F.2d 354, 371 (8th Cir. 1973) (test given to one applicant “consisted of four or five mathematical problems which a Company employee jotted down on a sheet of yellow paper”); Brito v. Zia Co., 478 F.2d 1200, 1205-06 (10th Cir. 1973) (test based almost entirely on subjective judgments of supervisors, not administered or scored under controlled and standardized conditions); Vulcan Society, supra, 490 F.2d at 396-98 (no job analysis, test measured abilities that were clearly of secondary importance to job). Exam No. 8155, in contrast, is a “second generation” selection procedure. Despite the various flaws in construction of the test, it is clear that some attempt was made to develop the test with recognition of at least some of the standards that courts had established in the first wave of Title VII cases. Aware that the validity of the test would likely have to be demonstrated, the City performed an extensive job analysis, consciously used Guideline concepts in determining the qualities that were being tested for, and attempted to eliminate extraneous variables, such as the applicant’s prior knowledge, his reading level, and his ability to complete the test in a relatively short amount of time. Nevertheless the plaintiffs have alleged and the District Court has concluded that the construction and use of Exam No. 8155 failed in several respects to meet test validity standards, particularly those specified in the Guidelines. Whether or not these deficiencies are fatal, they are plainly more substantial than the defects deemed not to defeat validity in prior cases. Detroit Police Officers Association v. Young, 446 F.Supp. 979, 990-91, 1007-08 (E.D.Mich. 1978); Bridgeport Guardians v. Bridgeport Police Department, 431 F.Supp. 931 (D.Conn.1977); cf. Washington v. Davis, 426 U.S. 229, 248-52, 96 S.Ct. 2040, 2051-53, 48 L.Ed.2d 597 (1976) (Fourteenth Amendment case involving some Title VII concepts). Consequently, assessment of Exam No. 8155 necessarily carries this Court into difficult areas of judging test validity. We must determine, with some care, what the general standards are for judging validity, and how these standards are to be applied in a specific factual situation. The study of employment testing, although it has necessarily been adopted by the law as a result of Title VII and related statutes, is not primarily a legal subject. It is part of the general field of educational and industrial psychology, and possesses its own methodology, its own body of research, its own experts, and its own terminology. The translation of a technical study such as this into a set of legal principles requires a clear awareness of the limits of both testing and law. It would be entirely inappropriate for the law to ignore what has been learned about employment testing in assessing the validity of these tests. At the same time, the science of testing is not as precise as physics or chemistry, nor its conclusions as provable. While courts should draw upon the findings of experts in the field of testing, they should not hesitate to subject these findings to both the scrutiny of reason and the guidance of Congressional intent. The need to modify rigid technical conclusions from the field of testing is indicated by the view of certain testing experts, including those who testified for the plaintiffs in this case, that there is no test that can be considered completely valid to select candidates for any but the most rudimentary tasks. If this view guided interpretation of Title VII, then at the current stage of the technology of testing, no test that produces a disparate racial impact could be used for positions such as police officers. While this position is a conceivable one, it is supported neither by the statutory language nor by judicial precedent. Had Congress felt that testing for virtually all employment was invalid, it would not have made a specific exception to Title VII for the proper use of professionally designed employment tests. Clearly, Congress did not intend that the standard used in interpreting Title VII would reject every test with a disparate racial impact. Nor have the courts permitted this to occur; although they have subjected employment testing to the careful scrutiny required by the statute, they have found a variety of tests to be valid, despite a disparate racial impact. See, e. g., Sims v. Sheet Metal Workers, Local 65, 489 F.2d 1023, 1025-26 (6th Cir. 1973); Detroit Police Officers Association v. Young, supra, 446 F.Supp. at 1007-08; Friend v. Leidinger, 446 F.Supp. 361 (E.D. Va.1977), aff’d, 588 F.2d 61 (4th Cir. 1978); United States v. South Carolina, 445 F.Supp. 1094 (D.S.C1977), aff’d, 434 U.S. 1026, 98 S.Ct. 756, 54 L.Ed.2d 777 (1978); Bridgeport Guardians v. Bridgeport Police Department, supra, 431 F.Supp. at 936-39; Jackson v. Nassau County Civil Service Commission, 424 F.Supp. 1162 (E.D.N.Y. 1976); Buckner v. Goodyear Tire & Rubber Co., 339 F.Supp. 1108, 1113-16 (N.D.Ala. 1972), aff’d, 476 F.2d 1287 (5th Cir. 1973). The danger of too rigid an application of technical testing principles is that tests for all but the most mundane tasks would lack sufficient validity to permit their use. At least that is the risk given the current state of the art of employment testing. This risk can be appreciated by considering the one example even most test critics acknowledge to have substantial, though not complete, validity. This is a typing test given to a group of applicants for jobs as typists. Such a test substantially meets all the eriteria suggested by plaintiffs’ experts for content validation, but the very success of this test casts doubt on the usefulness of the example. To begin with, typing is a task that readily yields to quantitative measurement. The quality of a typist’s job performance depends on two factors, both’ of which can be captured with precision in numbers: how fast he types, and how many errors he commits. Most jobs involve tasks whose performance can be evaluated only in the more subjective light of judgment. Surely this is true of nearly all the tasks required to be performed by police officers. In addition, there is a more basic problem with the typing test example. Typing is one of the few activities that a test-taker can perform in virtually the same manner as he will be required to perform on the job. That is obviously an ideal testing situation, but it is not one that is frequently available, and such “on-the-job” testing could not possibly be done to select police officers. Yet the force of the typing test example easily leads to one of the conclusions of the District Court in this case: that Exam No. 8155 lacked validity because it measured performance in an artificial classroom setting and did not necessarily indicate who would perform well on the job. Closely related to the question of the proper weight to be given to technical conclusions of testing theory is the question of the proper weight to be given to the EEOC Uniform Guidelines, which are largely based on these technical conclusions. See Guidelines § 5(C). The District Court drew its methodology from the Guidelines, concluding that the City’s test was invalid because it failed to satisfy all of the Guidelines. The Supreme Court has relied upon some of the Guidelines in several of the leading cases, see Albemarle, supra, 422 U.S. at 431, 95 S.Ct. at 2378; Espinoza v. Farah Manufacturing Co., 414 U.S. 86, 94, 94 S.Ct. 334, 339, 38 L.Ed.2d 287 (1973); Griggs, supra, 401 U.S. at 433-34, 91 S.Ct. at 854-55, but the Court has not ruled that every deviation from any of the Guidelines automatically results in a violation of Title VII. The Court appears to have applied the Guidelines only to the extent that they are useful, in the particular setting of the case under consideration, for advancing the basic purposes of Title VII. See Espinoza, supra, 414 U.S. at 94, 94 S.Ct. at 339; Guardians Association v. Civil Service Commission, 490 F.2d 400, 403 n.1 (2d Cir. 1973), United States v. Georgia Power Co., 474 F.2d 906, 913 (5th Cir. 1973). To the extent that the Guidelines reflect expert, but non-judicial opinion, they must be applied by courts with the same combination of deference and wariness that characterizes the proper use of expert opinion in general. See Albemarle, supra, 422 U.S. at 449, 95 S.Ct. at 2390 (Blackmun, J., concurring) (Guidelines “have never been subjected to the test of adversary comment. Nor are the theories on which the Guidelines are based beyond dispute.”) Thus, the Guidelines should always be considered, but they should not be regarded as conclusive unless reason and statutory interpretation support their conclusions. As this Court has previously stated: “If the EEOC’s interpretations go beyond congressional intent, the Guidelines must give way.” Guardians Association, supra, 490 F.2d at 403 n.1. In addition to their force as the expression of expert opinion, the Guidelines also possess legal force. But here too, it is necessary to keep their limits in mind. The primary purpose of the Guidelines is to indicate the standards that various Federal agencies, such as the EEOC, the Civil Service Commission, and the Department of Justice are to use in enforcing Title VII and related statutes. See Guidelines § 2(A). But the fact that an agency or group of agencies has announced the standards they will use does not convert those standards into mandatory legal rules. A second legal basis for following the Guidelines is that they represent the “administrative interpretation of the Act by the enforcing agency,” and are “entitled to great deference” on that basis. Griggs, supra, 401 U.S. at 433-34, 91 S.Ct. at 854-55; see Albemarle, supra, 422 U.S. at 431, 94 S.Ct. at 2378. However, the Court has also recognized that the Guidelines “are not administrative ‘regulations’ promulgated pursuant to formal procedures established by Congress.” Ibid. They are entitled to deference, not obedience. See Espinoza, supra, 414 U.S. at 94, 94 S.Ct. at 339 (1973) (Guideline rule on discrimination against non-citizen “is no doubt entitled to great deference . , but that deference must have limits where, as here, application of the guideline would be inconsistent with an obvious congressional intent”). Moreover, the Court in Griggs was following the Guidelines only to make the straightforward distinction between general intelligence tests and job-related tests; it is not at all clear that Griggs requires observance of all the intricate details of the Guidelines. It might be desirable for all employers to follow the more careful practices required of the Federal Government, but there is no reason to think that Congress intended to impose such practices, in their full rigor, when it enacted Title VII. With these considerations in mind, we turn to the validity of Exam No. 8155. IV. The Validity of Exam No. 8155 A. Selecting the Validation Technique The threshold task in determining the validity of a challenged examination is to select the appropriate method for assessing its job-relatedness. The Guidelines describe three techniques: content validation, construct validation, and criterion-related validation. Guidelines §§ 5(B), 14. The Guidelines specify when each technique is appropriate and also specify the requirements for successfully validating an exam by use of each technique. Defendants have attempted to justify Exam No. 8155 by content validation, a technique appropriate for tests that measure “knowledges, skills or abilities” representative of the “content” of the job. Guidelines § 14(C)(1). Plaintiffs contend that construct validation must be used to assess this exam because, in their view, the exam attempts to measure “constructs,” that is, inferences about mental processes or traits, such as “intelligence, aptitude, personality, commonsense, judgment, leadership and spatial ability.” Ibid. This content-construct distinction has a significance beyond just selecting the proper technique for validating the exam; it frequently determines who wins the lawsuit. Content validation is generally feasible while construct validation is frequently impossible. Even the Guidelines acknowledge that construct validation requires “an extensive and arduous effort.” Guidelines § 14(D)(1). The principal difficulty with construct validation is that it requires a technique that includes a criterion-related study, Guidelines § 14(D)(4)-a demonstration from empirical data that the test successfully predicts'job performance. Developing such data is difficult, and tests for which it is required have frequently been declared invalid. As a result, a conclusion that construct validation is required would often decide a case against a test-maker, once a disparate racial impact has been demonstrated. To determine whether defendants are entitled to use content validation, we examine the Guidelines’ criteria for that technique, but we do so bearing in mind our cautionary approach to the Guidelines, previously expressed. The Guidelines specify two basic conditions that must be met before content validation may be used. First, it must appear that what the test attempts to measure is knowledge or an ability, and not a general trait, such as intelligence. Guidelines § 14(C)(1). Second, the test must not measure knowledge or ability that an employee will be expected to learn on the job. Ibid.; see also § 5(F). The District Court rejected content validation, concluding both that Exam No. 8155 measures constructs, not abilities, and that, even if what was tested for could be considered abilities, they could be learned in the five-month training program. In specifying how the selection of validation techniques is to be made, the Guidelines adopt too rigid an approach, one that is inconsistent with Title VII’s endorsement of professionally developed tests. Taken literally, the Guidelines would mean that any test for a job that included a training period is almost inevitably doomed: if the attributes the test attempts to measure are too general, they are likely to be regarded as constructs, in which event validation is usually too difficult to be successful; if the attributes are fairly specific, they are likely to be appropriate for content validation, but this too will prove unsuccessful because the specific attributes will usually be learned in a training program or on the job. The origin of this dilemma is not any inherent defect in testing, but rather the Guidelines’ definition of “content.” This definition makes too sharp a distinction between “content” and “construct,” while at the same time blurring the distinction between the two components of “content”: knowledge and ability. The knowledge covered by the concept of “content” generally mean factual information. The abilities refer to a person’s capacity to carry out a particular function, once the necessary information is supplied. Unless the ability requires virtually no thinking, the “ability” aspect of “content” is not closely related to the “knowledge” aspect of “content”; instead it bears a closer relationship to a “construct.” Some researchers regard content tests as nothing more than assessments of particular kinds of constructs, e. g., Tenopyr, Content-Construct Confusion, 30 Personnel Psych. 47 (1977); others regard any ability that is evidenced by observable behavior as sufficiently non-inferential to be considered content, see Ebel, Comments on Some Problems of Employment Testing, 30 Personnel Psych. 55 (1977). See generally Catell, Validity and Reliability: A Proposed More Basic Set of Concepts, 55 J.Ed.Psych. 1 (1964). Whichever view is adopted, it would seem that abilities, at least those that require any thinking, and constructs are simply different segments along a continuum reflecting a person’s capacity to perform various categories of tasks. This continuum starts with precise capacities and extends to increasingly abstract ones-from the capacity for filling out forms to the capacity for exercising judgment. Recognition that abilities and constructs are not entirely distinct leads to a conclusion that a validation technique for purposes of determining Title VII compliance can best be selected by a functional approach that focuses on the nature of the job. The crucial question under Title VII is job relatedness-whether or not the abilities being tested for are those that can be determined by direct, verifiable observation to be required or desirable for the job. See Griggs, supra, 401 U.S. at 431, 91 S.Ct. at 853; Vulcan Society, supra, 490 F.2d at 394-95; Chance, supra, 458 F.2d at 1177. If the job in question involves primarily abilities that are somewhat abstract, content validation should not be rejected simply because these abilities could be categorized as constructs. However, if the test attempts to measure general qualities such as intelligence or commonsense, which are no more relevant to the job in question than to any other job, then insistence on the rigorous standards of construct validation is needed. Since tests of this kind are often biased in favor of a person’s familiarity with the dominant culture, permitting them to be used without a showing of predictive validity would perpetuate the effects of pri- or discrimination. But as long as the abilities that the test attempts to measure are no more abstract than necessary, that is, as long as they are the most observable abilities of significance to the particular job in question, content validation should be available. To lessen the risks of perpetuating cultural disadvantages, the degree to which content validation must be demonstrated should increase as the abilities tested for become more abstract. This functional approach, which adjusts the distinction between content and construct to the nature of the job being tested for, expands the opportunity for both employers and courts to rely on content validation. It also avoids making a threshold choice between content and construct validation based solely on the nature of the quality tested for unrelated to the job, a choice that might make content validation seem inappropriate. To base the content-construct determination on the nature of the job, it is necessary first to analyze the job to see if it requires abilities appropriate for content validation. Instead of choosing between content and construct validation at the outset, as the Guidelines seem to require, employers and courts can start the content validation inquiry and use its results to determine both whether content validation is appropriate and whether it has been achieved. Should the attempted content validation be found inadequate, the reason may be that this method of validation was not appropriate because of the pertinent job abilities revealed by the job analysis. On the other hand, this approach will sometimes indicate that content validation is appropriate, even though the abilities tested for could be considered constructs. Just as lessening the severity of the Guidelines’ distinction between content and construct reduces the likelihood that a test is invalid because it measures constructs, so sharpening the distinction between knowledge and ability, now obscured by the Guidelines, reduces the problem that the test is invalid because it duplicates the training period, i. e., tests for what will later be learned. Unlike knowledge, some abilities are appropriate for testing confirmed by content validation despite their overlap with post-selection training. A valid measurement of some abilities can select applicants who will ultimately use their training to perform their tasks more effectively or who will more effectively perform similar tasks for which they have not been specifically trained. On the other hand, content validation remains inappropriate for tests that measure knowledge of factual information if that knowledge will be fully acquired in a training program. Approval of such tests, without predictive validation, risks favoring applicants with prior exposure to the information, a course likely to discriminate against a disadvantaged minority. For example, it would be duplicative of the Police Department’s training program, and thus invalid, to test applicants for their knowledge of the Department’s arrest form. Testing for their ability to fill out the form, however, can be expected to select applicants who can be successfully trained to perform well at that task and others like it. Applying the approach just outlined, we conclude, at least as an initial matter, that content validation, may properly be selected as the appropriate technique for assessing Exam No. 8155. The exam tests for three basic abilities (although it purports to test for five): the ability to remember details, the ability to fill out forms, and the ability to apply general principles to specific facts. This third ability is assessed in three contexts: the application of general statements of criminal offenses to the facts of specific events, the application of procedures and standards to the facts of specific policing activities, and the application of procedures and standards to the facts of specific situations involving human relations problems. These three basic abilities are not so abstract, on their face, as to preclude content validation, provided subsequent consideration of the job analysis does not demonstrate that important and more concrete abilities necessary for the job were needlessly omitted from those considered for measurement. Though all three abilities involve some inference about mental processes, they are based on observable behaviors and are far less abstract than such traits as intelligence, leadership, or judgment. Moreover, testing for these three abilities sufficiently avoids the objection that the test duplicates the Department’s training program. Though all three abilities can be trained to some extent, the test-makers were entitled to select applicants with existing ability so that training would both enhance their abilities and prepare them for other tasks requiring similar talents. The vice of testing for knowledge readily taught in the training program was totally avoided. B. Assessing the Content Validity of Exam No. 8155 Since content validation appears to be an appropriate method for assessing Exam No. 8155, we proceed to consider whether the use of this method indicates that the exam has sufficient validity to select applicants for the job of police officer. The Guidelines describe various aspects of content validation, but do not neatly list ingredients of an adequate exam. From our study of the Guidelines, we distill five attributes of an exam with sufficient content validity to be used notwithstanding its disparate racial impact. The first two concern the quality of the test’s development: (1) the test-makers must have conducted a suitable job analysis, and (2) they must have used reasonable competence in constructing the test itself. The next three attributes are more in the nature of standards that the test, as produced and used, must be shown to have met. The basic requirement, really the essence of content validation, is (3) that the content of the test must be related to the content of the job. In addition, (4) the content of the test must be representative of the content of the job. Finally, the test must be used with (5) a scoring system that usefully selects from among the applicants those who can better perform the job. We consider- each of these five matters in turn. The Job Analysis According to the Guidelines, a job analysis involves an assessment “of the important work behavior(s) required for successful performance and their relative importance.” § 14(C)(2). The job analysis performed by the City, while somewhat flawed as the District Court pointed out, is nonetheless adequate to meet this standard. As far as the first part of the standard is concerned, the work behaviors involved in being a police officer were identified by extensive interviewing, and subjected to serious review (Job Analysis, Steps 1 and 2). The District Court found that these work behaviors “were not delineated with precision.” 484 F.Supp. at 795. In fact, the descriptions of the 42 tasks that ultimately appeared on the job analysis list vary considerably in the level of precision. Some are complete and unambiguous, such as “1. Checks the condition of personal and department equipment such as radio, patrol car, weapons, etc.”; “35. Attends training sessions.” Others are more open-ended, but do manage to fulfill their function by defining the behaviors associated with the task, such as “3. Performs foot patrol”; “40. Controls various types of crowds.” Still others are so vague that they communicate very little real information, such as “10. Interacts with juveniles in non-arrest situations”; “39. Performs duties in hostage situations.” While greater precision might have been achieved, a complete description of the observable tasks associated with being a police officer would be a reworded version of the entire training manual. The Police Department’s list of tasks, despite some lapses in specificity, contains a sufficient amount of meaningful information to satisfy the relevant requirement. The second part of the Guideline standard for a job analysis requires determination of the relative importance of the identified work behaviors. The City performed this function by means of an extensively distributed questionnaire, specifying the criteria to be used in ranking the 42 tasks (Job Analysis, Step 3). The process as a whole appears to be reasonably accurate, and neither the plaintiffs nor the District Court raised any serious objection to it. Having determined the work behaviors and established their relative importance, the City then grouped the 42 tasks into five clusters and asked panels of police officers to identify the knowledges, skills, or abilities necessary to the effective performance of these tasks. (Job Analysis, Steps 4 and 5). This function was implemented in a much less satisfactory manner. Only one of the panels identified the abilities; the other four used the list of five abilities that the first panel had developed. This lessened the value of having five independent panels make this complicated and subjective determination. Moreover, no effort was made to explain the relationship between any of the five abilities and the 42 job tasks from which they were ostensibly derived. The plaintiffs criticize the required abilities identified by the City for being undefined. But the type of definition suggested by the Guidelines-one that describes the abilities in terms of “observable behaviors and outcomes,” Guidelines § 15(C)(3)-seems repetitive, since the work behaviors are already defined in this way. The five identified abilities, with the possible exception of “human relations, including communication techniques,” are comprehensible enough. Their appropriateness for measurement would have been considerably clearer, however, if each panel had explained which tasks required which abilities. While the Guidelines may be unnecessarily stringent in regarding the identification of this relationship of ability to task as “essential,” see ibid., such identification does go far toward eliminating the ambiguities that are otherwise inherent in generalized descriptions of abilities. Only if the relationship of abilities to tasks is clearly set forth can there be confidence that the pertinent abilities have been selected for measurement. The Test Construction Process With a job analysis of questionable sufficiency, the City then proceeded to the test construction stage. As an initial matter, we note that Exam No. 8155 was developed “in-house," by staff members of New York City’s Police Department and Department of Personnel; there was little input from any outside source, and no participation by anyone specializing in test preparation. Of course, the law should not be designed to subsidize specialists. But employment testing is a task of sufficient difficulty to suggest that an employer dispenses with expert assistance at his peril. Certainly, the decision to forgo such assistance should require a Court to give the resulting test careful scrutiny. See Kirkland, supra, 520 F.2d at 425-26; Vulcan Society, supra, 490 F.2d at 395-96. While the determination of how many questions should be included for each identified ability was made by a fairly careful numerical analysis (Test Construction, Step 1), the process of writing the questions themselves was rather haphazard. The questions were initially framed by police officers, who may have had expertise in identifying tasks involved in their job but were amateurs in the art of test construction. In addition, the officers did not have access to the job analysis material during much of the process. Finally, the questions, although they were reviewed, were not tested on a sample population. To be sure, a complete determination of the questions’ accuracy in measuring the identified abilities would be equivalent in its complexity to a criterion-related study. But the City did not even perform the minimal sample testing to ensure that the questions were comprehensible and unambiguous. Not surprisingly, the test construction process did not fully succeed in meeting even its own goal of testing for all the identified abilities. As previously indicated, Exam No. 8155 does appear to test for the three identified abilities of remembering details, filling out forms, and applying general principles to specific facts. However, the fourth identified ability, human relations skill, proved more troublesome. In deciding how to test for this ability, the City faced a dilemma inherent in testing for all but the most mundane jobs. To be fully representative of the job, a test should measure all the significant abilities needed for successful job performance, yet some abilities, especially in jobs of any complexity, are far along the construct end of the content-construct continuum where successful validation is difficult. If a test tries to be representative and measure all significant abilities, including those that are clearly constructs, it risks the use of inadequate assessment devices, because the rigorous standard for construct validation will rarely be met. On the other hand, if the test-makers acknowledge the difficulty of satisfactorily measuring constructs and test only for those abilities that are appropriate for content validation, they encounter the objection that the test is not sufficiently representative of the job. Recognizing the difficulty of construct validation, yet reluctant to omit assessment of an important characteristic of successful job performance, the City attempted to resolve the dilemma by treating human relations skill as an ability suitable for content validation and devoting 30 questions, nearly one-third of the exam, to an effort to assess this ability. Mindful of an important requirement of content validity, the City carefully avoided rewarding a test-taker’s prior knowledge and, instead, supplied in the test itself all the information necessary to select the correct answers to the human relations questions. Included before each group of questions was a set of appropriate standards-essentially “do’s” and “don’ts”for handling a particular type of human relations matter. But supplying this guidance rendered the 30 questions primarily a further assessment of a candidate’s ability to apply written standards to specific fact situations, and only slightly a measure of his talent for human relations. Anyone with minimal analytic ability needed to apply the standards to the various fact situations could select the one correct answer, even if his intuitive reaction to a human relations problem might be woefully inadequate. Assessing human relations skill will always be a difficult enterprise, but the deficiency of the City’s attempt does not mean that a content validation approach is necessarily impermissible nor impossible to achieve. As indicated above, at least within the middle range of the content-construct continuum, the distinction between content and construct should be determined functionally, in relation to the job. If the quality measured is not unduly abstract, and if it constitutes a significant aspect of the job, content validation of the test component used to measure that quality should be permitted. But that component must be designed in an extremely careful way. Test-makers will be well advised to obtain highly qualified assistance in constructing this portion of an exam. One desirable approach would be to confront applicants with simulated real life situations and assess the appropriateness of their volunteered response. See Firefighters Institute for Racial Equality v. City of St. Louis, 616 F.2d 350 (8th Cir. 1980). That technique is normally too costly for large numbers of applicants, but might have usefulness as a testing device to be used toward the end of the overall selection procedure, after an initially large group of applicants has been narrowed down by the results of a written exam and a background check. If the test component is limited to traditional pencil and paper methods, it may be preferable to forgo any pretense of being able to make fine differentiation among candidates’ human relations skills and instead adopt a pass/fail approach, rejecting those whose demonstrably inappropriate responses to human relations questions mark them as unsuitable for police work. Another possibility is to recognize that questions in this area for which only one answer is correct are likely to be too easy, as were most of the questions on Exam No. 8155, and therefore of little use in making selections from among applicants. Instead questions can be designed for which some answers are appropriate responses and others are inappropriate. As feasible techniques in this area evolve, employers will be expected to use them. With these strengths and weaknesses of the job analysis and the test construction in mind, we now consider how well the test, as constructed and used, met the basic requirements of content validity. The Direct Relationship Requirement The central requirement of Title VII, relationship of test content to job content, was sufficiently satisfied by Exam No. 8155. The job analysis procedure provides adequate assurance that the identified tasks are in fact the tasks that a police officer performs. While the procedure for identifying the abilities required for those tasks was less satisfactory, the three abilities that were actually tested for appear adequately related to most of the identified tasks. The list of tasks confirms one’s intuitive assumption that police officers are required to fill out forms (see, e. g., “16. Processes arrests using appropriate police department forms and notifications”), to remember facts (see, e. g., “18. Gives testimony in court (oral and written)”), and to apply general principles to specific fact situations (see, e. g., “26. Executes warrants.”). Moreover, these abilities are among the most concrete ones that can be derived from the list; they are certainly more concrete than human relations skills, which the test purported to measure, but did not. Two of the abilities tested for, filling out forms and remembering facts, are as specifically stated as they could be without resort to trivial distinctions about particular kinds of forms and facts. The ability to apply general standards is somewhat more problematical, since it is a relatively abstract skill that is relevant to many jobs. However, if there is any job for which ability in applying and following rules is an especially important requirement, it is the job of a law enforcement officer. The Representativeness Requirement The second requirement established by the Guidelines is that the test must be a “representative sample of the content of the job.” As presented by the Guidelines, this representativeness requirement has two different meanings. The first is that the content of the test must be representative of the content of the job; the second is that the procedure, or methodology, of the test must be similar to the procedures required by the job itself. The Guidelines express this dual requirement in the following somewhat inscrutable language: “For any selection procedure measuring a knowledge, skill, or ability the user should show that (a) the selection procedure measures and is a representative sample of that knowledge, skill, or ability. . . . ” Guidelines § 14(C)(4) (emphasis added). Both aspects of the representativeness requirement, if interpreted rigorously, would once again foreclose any possibility of constructing a valid test. The United States, as amicus, argues that the requirement that the content of the exam be representative means that all the knowledges, skills, or abilities required for the job be tested for, each in its proper proportion. This is not even theoretically possible, since some of the required capacities cannot be tested for in any valid manner. Even if they could be, the task of identifying every capacity and determining its appropriate proportion is a practical impossibility. It is similarly impossible for the procedures of the test to be truly representative of the actual job procedures. Tests, by their nature, are a controlled, simplified version of the job activities, not the activities themselves. As a practical matter, virtually any realistic test, except one that directly measures a physical skill, like lifting 50-pound sacks, is likely to be a pencil and paper activity, quite different from the job it tests for. An elaborate effort to simulate the actual work setting would be beyond the resources of most employers, and perhaps beyond the capacities of even the most professional test-makers. More reasonable interpretations of the representativeness requirement are appropriate in light of Title VII’s basic purposes. The reason for a requirement that the content of the exam be representative is to prevent either the use of some minor aspect of the job as the basis for the selection procedure or the needless elimination of some significant part of the job’s requirements from the selection process entirely; this adds a quantitative element to the qualitative requirement-that the content of the test be related to the content of the job. Thus, it is reasonable to insist that the test measure important aspects of the job, at least those for which appropriate measurement is feasible, but not that it measure all aspects, regardless of significance, in their exact proportions. The reason for a requirement that the test’s procedure be representative is to prevent distorting effects that go beyond the inherent distortions present in any measuring instrument. For example, although all pencil and paper tests are dependent on reading, even if many aspe