Full opinion text
Opinion for the Court filed by Chief Judge WALD. WALD, Chief Judge: In this action, a class of women plaintiffs allege various forms of unlawful employment discrimination in the Foreign Service from 1976 to 1983. After a trial, the District Court found that no unlawful discrimination had occurred. See 616 F.Supp. 1540 (D.D.C.1985). This appeal followed. The record, however, discloses that the District Court’s decision was premised on errors of law and that several of its critical findings of fact were clearly erroneous. Consequently, we reverse, and remand for further proceedings in accordance with this opinion. I. Background Information A. The Foreign Service and Its Employment Practices The Foreign Service is our nation’s professional diplomatic corps. Members of the Service represent the interests of this nation abroad and assist the Secretary of State in the formulation of foreign policy at home. See 22 U.S.C. § 3904(1)-(1)2). The organization of Foreign Service personnel draws on the model of the United States military as well as the United States civil service. See S.Rep. No. 913, 96th Cong., 2d Sess. 2 (1980), U.S.Code Cong. & Admin. News 1980, P. 4419. For example, the Foreign Service is a “rank-in-person” system: members of the Service have an individualized rank which is independent of the rank of the particular job they happen to hold at any given time. H.R.Rep. No. 992, pt. 1, 96th Cong., 2d Sess. 3 (1980). The Foreign Service also copies the military in its “up or out” personnel system. Individuals must serve a probationary period of up to five years before they can receive a career appointment in the Service. 22 U.S.C. § 3946. If at the end of that period an individual has not received a career appointment, he or she must leave the Service. Id. § 3949. (Although according to the Foreign Service Act of 1980, the term “Foreign Service Officer” refers only to members of the Service with career appointments, and those serving under a limited, probationary appointment are called “career candidates,” the parties to this lawsuit use the term “Foreign Service Officer,” or “FSO,” to refer to those serving under both career and limited appointments. To avoid confusion, we will do likewise.) The Foreign Service assigns its officers to one of four areas of functional specialization, known as “cones”: political, economic, administrative, and consular. Officers in the political and economic cones deal with, respectively, political and economic dimensions to foreign relations and foreign policy. Officers in the administrative cone “are responsible for the support operations of U.S. embassies and consulates.” 616 F.Supp. at 1544 (115). Officers in the consular cone “work closely with the public providing assistance to American travelers and residents abroad, issuing visas [and dealing with] other immigration related issues.” Id. (116). As the District Court expressly found, the State Department does not encourage FSOs to change cones, and “[officers are expected to serve the major portion of their time in the Service” in the cones to which they were initially assigned. Id. (111110,14). Some officers, however, do switch cones. Senior FSOs who have demonstrated leadership ability may transfer into a “prestigious” program direction cone. Id. at 1554 (11104). Other FSOs are occasionally given temporary assignments to other cones or to some “inter-functional” positions. Id. at 1550 (H 70). Most FSOs applying to the Foreign Service at junior entry levels must take a written examination. Beginning in 1975, the examinations have tested applicants for aptitude in all four functional areas, and the Foreign Service has used the results of these examinations to determine a new FSO’s initial cone assignment. Id. at 1545 (1115.) A relatively small number of individuals have entered the Service laterally as mid-level FSOs. These lateral entrants bypassed the examination process and “selected, in advance, the functional field in which they wished to compete and were evaluated only for that specific cone.” Id. (H 17). Once in the Foreign Service, individuals change specific jobs frequently; the State Department has a policy of assigning individuals to positions for a set period of time, generally two to three years. See id. at 1550 (1171); H.Rep. No. 96-992, pt. 1, 96th Cong., 1st Sess. 3 (1980). Since 1975, job assignments in the Foreign Service have been made pursuant to an Open Assignment Policy, in which all members of the Service receive a list of vacant positions and submit “a bid list” indicating their preferences. These bid lists are compiled into a “bid book” from which assignment panels make their selections, after considering the interests and preferences of the bureau in which each position is located. Id. at 1550 (H1Í 73, 74). As previously indicated, some FSOs receive “out-of-cone” assignments pursuant to this process but in the main, job transfers are made inside the cones of initial assignment. In addition, FSOs do not necessarily receive a job position with a rank corresponding to the individual’s personal rank. Positions that have a higher rank than the individual are known as “stretch” assignments. Positions with a lower rank than the individual’s are “down-stretch” assignments. Pursuant to the Open Assignment Policy, individuals do not receive stretch or down-stretch assignments unless they bid for them, but as with any other assignment, individuals do not receive these assignments simply because they bid for them. Id. at 1551 (1177). The Foreign Service prepares annual written evaluations of its officers’ job performance. In addition to rating the actual past performances of FSO’s, the evaluations rate the potential of the FSOs future job performance. 616 F.Supp. at 1549. The State Department also gives out Honor Awards in recognition of outstanding achievement. In descending order of prestige are the Distinguished Honor Award, the Superior Honor Award, and the Meritorious Honor Award. See Plaintiffs’ Post-Trial Brief at 112-13. Except for Senior members, salaries in the Foreign Service are based on a schedule established by the President which consists of nine salary classes. 22 U.S.C. § 3963. The Secretary of State assigns all Foreign Service Officers to a particular salary class. Id. § 3964. By statute, except in limited circumstances, a career candidate for appointment as a Foreign Service Officer may not be initially assigned to a salary class higher than class 4 (class 1 being the highest). Id. § 3947. Usually career candidates are placed initially in class 7 or class 8. Promotions from one salary class to another are made by the Secretary of State after receiving recommendations and rankings submitted by selection boards which evaluate the members of each class. Foreign Service Officers do not compete for promotions until the transition from class 6 to class 5; until then, they are promoted at the end of an established time period if they perform their duties satisfactorily. See Joint Appendix (“J.A.”) at 117-121; Defendant’s Post-Trial Brief at 96. B. The History of This Litigation This class action began over ten years ago when appellants filed their complaint alleging that widespread discrimination against women in the Foreign Service violated Title VII of the Civil Rights Act of 1964, as amended in 1972 to cover employment discrimination in the federal government. See 42 U.S.C. § 2000e-16. The parties subsequently resolved by consent decree all claims relating to admission into the Foreign Service. The appellants’ claims of discriminatory personnel actions against women already in the Foreign Service proceeded to trial in the District Court. The parties agreed to try initially only the issue of liability, leaving appropriate remedies to a subsequent phase of the proceedings, if necessary. After trial on the liability issue, the District Court concluded that appellants “failed to show by a preponderance of the evidence any sexual discrimination by the State Department.” 616 F. Supp. at 1561. The court entered a final judgment for the Secretary of State, dismissing the complaint. Id. This appeal followed from the District Court’s failure to find sex discrimination in seven different types of personnel practices. First, the appellants claim that from 1976 to 1983, the Foreign Service discriminated against women in the initial cone assignments of entering FSOs; the State Department assigned proportionally fewer women than men to the political cone and proportionately more women than men to the consular cone. This disparity was allegedly caused by the differing scores of women and men on the Foreign Service entrance examinations, producing a disparate impact on women and men candidates in violation of Title VII. Second, women were given proportionally fewer out-of-cone assignments to the program direction cone and proportionally more out-of-cone assignments to the consular cone. Third, women were given proportionally fewer “stretch” assignments and proportionally more “downstretch” assignments than men in the same class. Fourth, women received a disproportionately low number of appointments as Deputy Chief of Mission, the position just below that of Ambassador. Fifth, in its evaluation reports, the State Department gave lower future potential ratings to women than men despite equivalent ratings for their past performance. Sixth, women received a disproportionately low number of Foreign Service Honor Awards. And seventh, the State Department promoted women from class 5 to class 4 at a lower rate than it promoted men. With respect to each of these seven personnel practices, the appellants offered data showing a disparity between men and women, along with a statistical analysis designed to demonstrate the improbability that a disparity of that scale could result from chance. The data and analysis, they allege, provide a strong basis for inferring that this disparity was the product of unlawful discrimination. In addition, the appellants introduced nonstatistical evidence pertaining generally to the existence of a prejudicial attitude towards women in the Foreign Service from 1976 to 1983. The District Court, however, rejected the inference of unlawful discrimination in each of the seven areas. In discounting the probative force of appellants’ statistics, the District Court said that their statistical studies rested on faulty data, or flawed methodology, or omitted a crucial variable that would explain the disparity between men and women in a nondiscriminatory way. The District Court also said that some of the statistical evidence focused on too narrow a segment of Foreign Service personnel practices. As we shall explain, the District Court’s treatment of the appellants’ evidence was in some instances contrary to law and in other respects clearly erroneous as a matter of fact. II. Title VII Claims: Two Different Theories Under Title VII a plaintiff can rely on either of two different theories to support a claim of unlawful sex discrimination. A “disparate treatment” claim alleges that the defendant intentionally based an employment decision on the sex of the plaintiffs. See, e.g., International Brotherhood of Teamsters v. United States, 431 U.S. 324, 335 & n. 15, 97 S.Ct. 1843, 1854 & n. 15, 52 L.Ed.2d 396 (1977). Disparate treatment claims can involve an isolated incident of discrimination against a single individual, or, as in this case, allegations of a “pattern or practice” of discrimination affecting an entire class of individuals. Id. A “disparate impact” claim alleges that the defendant based an employment decision on a criterion that although “facially neutral” nevertheless impermissibly disadvantaged individuals of one sex more than the other. Id. at 336 n. 15, 97 S.Ct. at 1854 n. 15. This case is a “classic” example of a disparate impact claim in which plaintiffs allege that the defendant based employment decisions on the results of a test for which members of one sex on average received lower scores than members of the other sex. See B. Schlei & P. Grossman, Employment Discrimination Law at 13 (1983-84 Supp.); see also Griggs v. Duke Power Co., 401 U.S. 424, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971) (the original disparate impact case). Because these two theories are distinct, we must consider them separately. Appellants’ only disparate impact claim concerns the initial cone assignments; the other six claims involve disparate treatment and we will consider them first. III. Legal Principles Applying to Pattern or Practice Disparate Treatment Claims In a typical sex discrimination pattern or practice disparate treatment case, plaintiffs allege the existence of a disparity between men and women in selection rates for a particular job or job benefit and further allege that this disparity was caused by an unlawful bias against members of the disadvantaged sex, usually women. To prevail in their claim, plaintiffs must prove, by a preponderance of the evidence, that these allegations are true. Proof of the disparity itself is based upon a comparison of the proportion of those women eligible for selection who were actually selected with the corresponding proportion of eligible men who were actually selected. Plaintiffs establish a disparity disfavoring women if the evidence demonstrates that the selection rate for eligible women was less than the selection rate for eligible men. Sometimes, the disparity is expressed as the difference between the number of women actually selected and the number of women one would expect to have been selected, assuming equality in the selection rates for men and women. (If one knows the number of women eligible and the selection rate for men, one can determine, using algebra, the expected number of successful women.) Proof that the observed disparity was caused by an unlawful bias against women need not be direct. Circumstantial evidence that the disparity, more likely than not, was a product of unlawful discrimination will suffice to prove a pattern or practice disparate treatment case. See Teamsters, 431 U.S. at 335 n. 15, 97 S.Ct. at 1854 n. 15. Indeed, this circumstantial evidence may itself be entirely statistical in nature. See, e.g., Segar v. Smith, 738 F.2d 1249, 1278-79 (D.C.Cir.1984), cert. denied sub. nom. Meese v. Segar, 471 U.S. 1115, 105 S.Ct. 2357, 86 L.Ed.2d 258 (1985). In this case, appellants rely to a great extent on statistical evidence to prove their claims of disparate treatment. We find it necessary, therefore, to discuss how statistical analysis of an observed disparity can raise an inference of unlawful discrimination. A. Raising An Inference of Discrimination With Statistical Evidence A disparity between the selection rates of men and women for a particular job or job benefit has one of three possible causes. See D. Baldus & J. Cole, Statistical Proof of Discrimination 291 (1980). First, the disparity may be a product of an unlawful discriminatory animus; this is what plaintiffs are attempting to prove. Second, the disparity may have a legitimate and nondiscriminatory cause. For example, prior experience of a certain type may be an important factor in making certain employment decisions, and if it happened to be true that women on the average have less of this experience than men, one would expect that women could be selected less frequently. Third, the disparity may simply be a product of chance. Even if we may properly assume that, as a general rule, women and men on average are equally qualified to be selected for a particular job or job benefit, for any particular group of men and women who happen to constitute the actual pool of eligible candidates at the time the selections are made, there may be some deviation from this general rule because the actual qualifications of men and women differ from individual to individual and any particular pool of eligible candidates constitutes an inherently random collection of individuals. Thus, even if selections were made entirely on the basis of qualification, without a trace of discriminatory bias, random deviations in the selection rates for men and women may result. A statistical analysis of a disparity in selection rates can reveal the probability that the disparity is merely a random deviation from perfectly equal selection rates. Statistics, however, cannot entirely rule out the possibility that chance caused the disparity. Nor can statistics determine, if chance is an unlikely explanation, whether the more probable cause was intentional discrimination or a legitimate nondiscriminatory factor in the selection process. See id. at 290-92. Title VII nevertheless provides that if the disparity between selection rates for men and women is sufficiently large so that the probability that the disparities resulted from chance is sufficiently small, then a court will infer from the numbers alone that, more likely than not, the disparity was a product of unlawful discrimination — unless the defendant can introduce evidence of a nondiscriminatory explanation for the disparity or can rebut the inference of discrimination in some other way. See Hazelwood School District v. United States, 433 U.S. 299, 307-08, 97 S.Ct. 2736, 2741, 53 L.Ed.2d 768 (1977) (“Where gross statistical disparities can be shown, they alone in a proper case constitute prima facie proof of a pattern or practice of discrimination.”); see also Segar, 738 F.2d at 1278 (“[W]hen a plaintiffs methodology focuses on the appropriate labor pool and generates evidence of [a disparity] at a statistically significant level,” this evidence alone will be “sufficient to support an inference of discrimination.”). The preliminary question for a court, then, is at what point is the disparity in selection rates is sufficiently large, or the probability that chance was the cause sufficiently low, for the numbers alone to establish a legitimate inference of discrimination. Although this question is crucial in Title VII litigation, the answers given by courts have been regrettably imprecise. The Supreme Court has twice stated that “[a]s a general rule for ... large samples, if the difference between the expected value and the observed number is greater than two or three standard deviations, then the hypothesis that [the disparity] was random would be suspect to a social scientist.” Castaneda v. Partida, 430 U.S. 482, 497 n. 17, 97 S.Ct. 1272, 1281, n. 17, 51 L.Ed.2d 498 (1977); see also Hazelwood, 433 U.S. at 309 n. 14, 97 S.Ct. at 2742 n. 14 (quoting Castaneda). But many lower courts and commentators have noted that the difference between two and three standard deviations is considerable and that, therefore, the Supreme Court’s statement falls short of establishing an exact legal threshold at which statistical evidence, standing alone, establishes an inference of discrimination. See, e.g., Segar, 738 F.2d at 1283 n. 28. This court, using different terminology, has stated that statistical evidence meeting “the .05 level of significance ... [is] certainly sufficient to support an inference of discrimination.” Segar, 738 F.2d at 1283. “[T]he .05 level,” the Segar opinion explained, “indicates that the odds are one in 20 that the result could have occurred by chance.” Id. at 1282. (This statement is somewhat imprecise and has predictably led to confusion, as we discuss infra.) The Segar court justified the consistency of its statement with the statements of the Supreme Court by observing that “[a] level of two standard deviations corresponds to statistical significance at the .05 level.” Id. at 1283 n. 28. In this case, the District Court cited Segar in its Conclusions of Law, stating: “The Court adopts the .05 level for establishing that a [statistical] study is statistically significant.” 616 F.Supp. at 1559 (if 14). But the District Court then went on to say that “[t]he .05 level generally corresponds to 1.65 standard deviations.” Id. How can a 5% probability of randomness correspond both to a measurement of two standard deviations and a measurement of 1.65 standard deviations, one may reasonably ask? There is a legitimate answer: it depends on whether one is using a “one-tailed” or a “two-tailed” test of statistical significance. A disparity measuring 1.65 standard deviations corresponds to a 5% probability of randomness under a one-tailed test. A disparity measuring two standard deviations (to be more precise, 1.96 standard deviations) corresponds to a 5% probability of randomness under a two-tailed test. This difference between one-tailed and two-tailed tests obviously requires further explanation. It also presages the obvious question, given the substantial differences in result, of which test is the more appropriate one to use in Title VII cases. Neither this court’s opinion in Segar nor the District Court's opinion in this case discusses the difference between “one-tailed” or “two-tailed” approaches. The Supreme Court has given us no explicit guidance on this issue. And, unfortunately, neither side to this litigation has devoted more than a single footnote each to this difficult but important issue. See Appellants’ Reply Brief at 32 n. 38; Appellee’s Brief at 62 n. 73. For obvious reasons we, too, confront this issue with some trepidation. But appellants’ and appellee’s evidence on the un-derpromotion of women from FSO class 5 to class 4 measures 1.88 and 1.76 standard deviations, respectively. (The difference results from the use of some different data. See 616 F.Supp. at 1557 (H130).) Whether one adopts the appellants’ or the appellees’ number as the better evidence, it falls between 1.65 and 1.95 standard deviations. Therefore, if one tests the statistical significance of this number using the Se-gar standard of a 5% probability of randomness, the outcome turns on whether one uses a one-tailed or two-tailed test. Under a one-tailed test, the number is statistically significant (because it is larger than 1.65 standard deviations, which correspondents to a 5% probability of randomness under a one-tailed test) and therefore by itself establishes a prima facie case of disparate treatment. Under a two-tailed test, the number does not quite reach the statistically significant threshold (because it is smaller than 1.96 standard deviations, which corresponds to a 5% probability of randomness using a two-tailed test) and therefore by itself does not raise an inference of discrimination. Given the unavoidability of embarking upon a journey into the statistical maze, we begin with the terms “one-tailed” and “two-tailed”; they refer to the “tails” or ends of the bell-shape curve, which represents in graph form a “random normal distribution.” E.g., W. Curtis, Statistical Concepts for Attorneys 72-73 (1983); see Diagram 1 copied from id. In these random distributions, the area under any segment of the bell curve measures the probability of that range of results occurring randomly. Id. Furthermore, the percentage area underneath the bell curve within one standard deviation (<r) distance from the mean (p) of a normal distribution is always the same for all normal distributions (regardless of the specific value of a or p, or the units in which these terms are measured). Thus, the probability of a result randomly occurring that measures within one standard deviation of the mean of the distribution (either greater or lesser than the mean) is the same for all normal distributions: 68.26%. Id. Indeed, this relationship holds true for any distance from the mean, measured in numbers of standard deviations. For example, the probability of a result occurring within two standard deviations from the mean is 95.44% and the probability of a result occurring within three standard deviations is 99.73%. See Diagram 1. Thus, for all normal distributions, the probability of randomness is directly associated with a measurement in numbers of standard deviations. Diagram 1 But for every deviation from the mean of a normal distribution, measured in a certain number of standard deviations, there are two distinct ways of referring to the probability of that result occurring randomly. For example, if fewer women than expécted were selected for a particular job, and this disparity measured 2.17 standard deviations, we can ascertain the probability that women by chance would be underse-lected to this extent or greater. This probability corresponds to the area between 2.17 standard deviations and the end of the bell curve representing the most extreme underselection of women. Standard statistical tables reveal that this probability is only 1.5%. See B. Lindgren & D. Berry, Elementary Statistics 479 (1981). We can speak of the probability measurement associated with 2.17 standard deviations in another way, however. Although the observed disparity between the actual and expected number of women in this example was an underselection of women, there is a corresponding possibility that women might randomly be overselected such that the difference between the expected number of women selected and the number of women selected due to this random overselection also measures 2.17 standard deviations. The probability of a random deviation from the expected number of women selected with a magnitude of 2.17 standard deviations or larger, resulting from either an underselection or overselection of women, corresponds to the area under the bell curve between 2.17 standard deviations and both extremes of the curves: 3%. The difference between “one-tailed” and “two-tailed” tests of statistical significance stem from these two different ways of measuring probability. If one decides (as the Segar court did) to reject the hypothesis that an observed disparity from an expected result occurred randomly only if the observed disparity falls within the range of the 5% most extreme possible disparities, one must still decide whether the 5% range should be entirely within only one of the tails of the bell curve, or instead should be divided with half of the range in each tail. Five percent of the total bell curve can be found either in the range from 1.65 standard deviations from the mean to one extreme end of the bell curve or in the area from 1.96 standard deviations to both extreme ends of the bell curve. Compare Diagrams 2 and 3, copied from V. Cangelo-si, P. Taylor & P. Rice, Basic Statistics 173-74 (1979). For this reason, a 5% probability of randomness corresponds to 1.65 or 1.96 standard deviations, depending upon whether one uses a one-tailed or a two-tailed test. (Similarly, 1.65 standard deviations correspond to a 10% probability of randomness under a two-tailed test; and 1.96 standard deviations correspond to a 2.5% probability of randomness under a one-tailed test.) Diagram. 2 Diagram S We are now, hopefully, in a position to address whether in a Title YII case, a court should use a one-tailed or two-tailed test to determine whether statistical evidence alone should raise an inference of unlawful discrimination, recognizing that there is a difference of opinion among courts and commentators on the issue. Compare, e.g., EEOC v. Federal Reserve Bank of Richmond, 698 F.2d 633 (4th Cir.1983), rev’d on other grounds sub. nom. Cooper v. Federal Reserve Bank of Richmond, 467 U.S. 867, 104 S.Ct. 2794, 81 L.Ed.2d 718 (1984), with Little v. Master-Bilt Products, Inc., 506 F.Supp. 319 (N.D.Miss.1980). Indeed, one leading treatise on the role of statistical evidence in Title VII litigation has shifted its position between the publication of the main text and the publication of a supplement. In the main text of their book, Baldus and Cole write: [S]tatistical texts frequently recommend the use of a one-tailed test when the only question of interest is the likelihood of a difference in one direction, e.g., when only a positive disparity between two numbers is of interest. This practice supports the use of a one-tailed test in discrimination cases, since the issue is always whether one group is favored over another. A defendant will argue, however, that both minority and majority groups [or men and women] are protected from discrimination and it is therefore inequitable to disregard the probability of outcomes that may favor either group. Since there is no clear answer to this question, the most desirable approach is an awareness of the conceptual and practical differences between the two types of tests and a consistent use of the same type of test in similar cases whenever practical. We have used two-tailed tests throughout this book. D. Baldus & J. Cole, Statistical Proof of Discrimination 307-08 (1980) (footnote omitted). In the most recent supplement, however, the authors criticize as “unnecessarily strict” the Fourth Circuit’s decision in EEOC v. Federal Reserve Bank of Richmond to require a two-tailed approach unless “independent evidence indicates the presence of discrimination of the type being challenged.” D. Baldus & J. Cole, Statistical Proof of Discrimination 129 (1986 Cumulative Supp.) (footnote omitted). Bal-dus and Cole then state a preference for a legal rule that would allow a one-tailed test “if the possibility of intentional discrimination favoring the protected group represented by plaintiff [e.g., women in this case] can be ruled out as defying logic, i.e., the available evidence excluding the statistic in question gives strong support to the conclusion that the system is either nondiscriminatory or disadvantageous to the plaintiff’s group.” Id. at 129-30. In a footnote to this passage, the authors continue: The logic underlying this statement is that if one can be certain that there was no discrimination in favor of plaintiff’s group, then any disproportionate impact would simply be interpreted as being a chance outcome in an equitable process. Id. at 130 n. 38. Although the latest position adopted by Baldus and Cole makes some sense, we reject its applicability to the present case. We note that some of appellants’ claims of unlawful discrimination involved complaints that women were overselected for particular kinds of jobs, e.g., consular cone and downstretch assignments. Appellants undoubtedly have the right under Title VII to object to the State Department’s selection of FSOs for these positions on the basis of sex. Such claims of discriminatory overselection, however, require a two-tailed statistical analysis. Appellants may view consular assignments as inferior to political assignments, but another class of women plaintiffs could certainly bring a Title VII claim if women were intentionally underas-signed to the consular cone. Consequently, statistically significant deviations in either direction from an equality in selection rates would constitute a prima facie case of unlawful discrimination. Indeed, appellants’ own statistical expert testified that a two-tailed test was necessary in evaluating the disparity between men and women in assignments to the consular cone because the hypothesis to be tested is whether cone assignments are made without regard to sex. See Transcript (Tr.) at 1081. We also think a two-tailed test of statistical significance should be applied to all of appellants’ discrimination claims in this case. First, Baldus and Cole originally noted the importance of consistency in evaluating statistical evidence. Second, although we by no means intend entirely to foreclose the use of one-tailed tests, we think that generally two-tailed tests are more appropriate in Title VII cases. After all, the hypothesis to be tested in any disparate treatment claim should generally be that the selection process treated men and women equally, not that the selection process treated women at least as well as or better than men. Two-tailed tests are used where the hypothesis to be rejected is that certain proportions are equal and not that one proportion is equal to or greater than the other proportion. See Curtis, supra, at 119-22, 133-37. Moreover, even if a disparity in only one direction is at issue in a particular Title VII case (e.g., only the underpromotion and not the overpromotion of women), we think that the more appropriate assessment of the probability that the contested disparity resulted from chance requires a recognition that a random disparity of equal magnitude, but in the opposite direction, is equally as likely. For example, if plaintiffs in a Title VII case come into court simply with evidence that women were underselected for a particular job, and that this disparity measured 1.75 standard deviations, it is perfectly true that the probability of women being underselected to this extent or more by chance is only 4%. Under a one-tailed test of statistical significance, employing the 5% level, as this court did in Segar, this evidence alone would establish a prima facie case of disparate treatment. But for a disparity measuring 1.75 standard deviations it is equally true that the probability of a random deviation of this magnitude or larger, either underselecting or overselecting women, is 8%. In other words, disparities of this magnitude will be consistent with the hypothesis that the selection process did not treat men and women differently in 8% of the cases. Even if in the case before the court the disparity disfavors women and not men, how can the court ignore the possibility that the case might still be one of the 8% cases in which a fair selection process would by chance produce disparities in this magnitude or greater? Thus, we think a court should generally adopt a two-tailed approach to evaluating the probability that the contested disparity resulted by chance. Furthermore, although an 8% probability is pretty low, we do not think that it is low enough to establish by itself an inference of unlawful discriminatory animus. We think that statistical evidence must meet the 5% level referred to in Segar for it alone to establish a prima facie case under Title VII. Taken together, as we have said, a two-tailed test and a 5% probability of randomness require statistical evidence measuring 1.96 standard deviations. Consequently, if plaintiffs come into court relying only on evidence that the underselection of women for a particular job measured 1.75 standard deviations, it seems improper for a court to establish an inference of disparate treatment on the basis of this evidence alone. Of course, plaintiffs in Title VII pattern and practice cases need not rely on statistical evidence alone. Because the ultimate issue in a disparate treatment case is whether the disparity resulted from unlawful discriminatory animus, plaintiffs may introduce any additional evidence which is probative on this issue. Thus, plaintiffs are in no way foreclosed from establishing an inference of discrimination simply because the contested disparity falls short of the 1.96 standard deviations mark when analyzed statistically. Obviously, to use an extreme example, if an employer admits under cross-examination that assignments for a certain position were based in large part on sex, it matters not that the observed underselection of women measures only 1.75 standard deviations. When plaintiffs in a Title VII pattern or practice case rely on evidence in addition to the evidence of the disparity itself, the issue for the trier of fact in determining whether the plaintiffs have established a prima facie case must be whether the totality of plaintiffs’ evidence (again including the evidence of the disparity itself) demonstrates that, more likely than not, the disparity resulted from an unlawful discriminatory animus— just as the issue after all the relevant evidence has been introduced by both sides remains whether in light of the totality of the evidence, plaintiffs have shown that, more likely than not, the disparity resulted from discrimination. B. The Applicability of Title VII to Any Personnel Action A plaintiff may bring a Title VII claim for alleged discrimination with respect to any employment decision by an agency of the federal government. The statute itself states that “all personnel actions affecting employees or applicants for employment ... shall be made free from any discrimination based on ... sex.” 42 U.S.C. § 2000e-16. In the Foreign Service Act of 1980, Congress reiterated this requirement specifically for Foreign Service employment practices. 22 U.S.C. § 3905. Moreover, in the 1980 Act, Congress specifically defined a “personnel action,” which must be free from sex discrimination, to encompass “(A) any appointment, promotion, assignment (including assignment to any position or salary class), award of performance pay or special differential, within-class salary increase, separation, or performance evaluation and (B) any decision, recommendation, examination, or ranking provided for under this chapter which relates to any action referred to in subparagraph (A).” Id. This language could hardly be more inclusive. From this statutory language, two legal principles necessarily follow. First, appellants in this case may bring a disparate treatment claim regarding discrimination in any type of personnel decision regardless of whether or not that discrimination has an effect on other, arguably more important, personnel decisions. Thus, if the State Department has intentionally discriminated against women in certain types of assignment decisions, the State Department has violated 42 U.S.C. § 2000e-16 even if the State Department can prove that the unlawful discrimination in assignments did not adversely affect the opportunities of women for promotion in the Foreign Service. It is beyond dispute that the State Department may not discriminate against women in making any kind of employment decision, and if the State Department breaches this requirement, appellants have a cause of action to vindicate their statutory rights. We note, as further support of our interpretation of 42 U.S.C. § 2000e-16, that the Supreme Court last Term interpreted an analogous Title VII provision applying to private employers to encompass a claim of sex discrimination for sexual harassment even if the sexual harassment caused no tangible or economic loss. Meritor Savings Bank, FSB v. Vinson, — U.S. -, 106 S.Ct. 2399, 91 L.Ed.2d 49 (1986). The provision of Title VII involved in Vinson makes it “an unlawful employment practice for an employer ... to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment, because of such individual’s ... sex.” 42 U.S.C. § 2000e-2(a)(l). The language of 42 U.S.C. § 2000e-16, involved here, is even broader, covering “all personnel actions” based on sex, regardless of whether the personnel action affects promotions or causes other tangible or economic loss. Second, and relatedly, if plaintiffs in a Title VII case claim discrimination in certain kinds of employment decisions, it is no defense that the government did not discriminate against women in other kinds of employment decisions. For example, if the State Department intentionally under-selected women for appointment as Deputy Chiefs of Mission (DCM), the State Department has violated 42 U.S.C. § 2000e-16 even if the State Department can prove that it did not discriminate against women in assignments to five other “high visibility” positions. Appellants need not allege or prove discrimination in assignments to other “high visibility” positions in order to maintain a cause of action with respect to discrimination in DCM assignments. As the Supreme Court has stated: “Of course, Title VII provides for equal opportunity to compete for any job.” Teamsters, 431 U.S. at 338 n. 18, 97 S.Ct. at 1856 n. 18 (emphasis in original). Although under 42 U.S.C. § 2000e-16 appellants must not be required to prove discrimination in employment decisions other than the ones they are specifically contesting, the government is correct in arguing that evidence of nondiscrimination in those other employment decisions may be probative of whether intentional discrimination actually occurred in the contested employment decisions. For example, if an employer can demonstrate that it did not discriminate against women at several steps of a promotional ladder, that evidence, in some circumstances, may reasonably suggest that the employer did not discriminate in the step at issue either. But courts must be especially careful in judging the relevance of this kind of evidence lest they contravene the legal rule that under 42 U.S.C. § 2000e-16 plaintiffs need not prove discrimination in personnel actions other than those specifically at issue. The evidence supporting an inference of unlawful discrimination in certain employment decisions may be sufficiently strong that evidence of nondiscrimination in other employment decisions cannot rebut this inference. Thus, in some cases the strength of appellants’ prima facie case is so great that even if they were to agree to a stipulation that sex discrimination did not occur in other employment decisions, their evidence as to the employment decisions specifically at issue would still prove that, more likely than not, unlawful discrimination occurred. When all the evidence raising and rebutting the inference of discrimination is statistical, according the proper deference to each legal principle is a delicate task indeed. If Title VII plaintiffs are able to muster only the most marginal inference of discrimination in only one type of job decision (e.g., the underselection of women in one promotional class measures only 1.98 standard deviations), then an inference of discrimination may be undercut by the fact that women are demonstrably not underse-lected in other similar job decisions. But even here courts must be wary. Evidence that the underselection of women in another similar job decision measures just below the 1.96 threshold, while not sufficient to prove discrimination, is not compelling evidence that the employer did not discriminate in this other employment decision. Thus, when plaintiffs in a Title VII case introduce statistical evidence of an extreme disparity in the selection rates for men and women for a certain type of job, the fact that these plaintiffs have insufficient evidence to establish an inference of discrimination regarding other employment decisions should not block an inference of discrimination on the specific type of employment decision at issue. For example, if Title VII plaintiffs present evidence that the underselection of women for a particular type of job assignment measures above 3.0 standard deviations, this evidence necessarily raises an inference of discrimination in these assignments regardless of the statistical evidence concerning other assignments. The likelihood that this disparity in the selection rate for men and women is merely a random deviation in a selection process that treated men and women equally is simply too low (l-in-500 using a two-tailed approach) for statistical evidence regarding other assignment decisions to rebut this evidence. In these circumstances, the Title VII defendant must present evidence directly relating to the type of assignment at issue to explain the evident disparity in a legitimate, nondiscriminatory fashion. For a district court to reject plaintiffs’ claim of discrimination in such a case on the grounds that plaintiffs failed to raise an inference of discrimination in other job assignments would effectively amount to a requirement that plaintiffs prove discrimination in employment decisions other than those specifically at issue. And, as we have said, such a requirement would directly conflict with the express provisions of 42 U.S.C. § 2000e-16. C. Rebutting the Inference of Disparate Treatment As we have discussed, under Title VII courts will initially infer that a disparity between men and women in selection rates for a particular job or job assignment results from unlawful discrimination if the disparity is large enough: i.e., measures at least 1.96 standard deviations. But defendants in Title VII cases must be offered an opportunity to rebut this inference by showing that the disparity, albeit nonrandom in cause, resulted from some legitimate, nondiscriminatory factor. Similarly, defendants must be allowed to rebut the inference of discrimination by, alternatively, challenging the statistical calculations upon which the inference of discrimination is based. For example, the statistics may rely on faulty data, flawed computations, or improper methodologies. A recent Supreme Court opinion provides courts with some guidance on how to treat attempts to attack an inference of discrimination based on statistical evidence alone. See Baze-more v. Friday, — U.S. -, 106 S.Ct. 3000, 92 L.Ed.2d 315 (1986). In Bazemore, the United States District Court for the Eastern District of North Carolina was presented with statistical evidence that black employees of the North Carolina Agricultural Extension Service received substantially lower salaries than white employees working in the same job positions. The District Court determined that “the statistical evidence of plaintiffs standing alone and without further explanation probably suffices to make out a prima facie showing of discrimination in salaries.” Civil Action No. 2879, Mem. Op. at 47 (August 22, 1982). The defendants in Bazemore, however, argued that plaintiffs’ statistics failed to account for several factors, any of which would provide a legitimate, nondiscriminatory explanation for the salary disparities. Id. at 48. The District Court agreed with the defendants, holding that because defendants had demonstrated that these other factors might have caused the salary disparities, defendants successfully rebutted plaintiffs’ inference of disparate treatment: Having thoroughly considered all of the evidence bearing on the salary issue and the contentions of the parties based thereon, the court has concluded that if it be assumed that plaintiffs made out a prima facie case on this issue, it has only been by virtue of the plaintiffs’ statistical evidence ...; that because of their failure to include many of the vital factors necessary to be considered in fixing salaries the probative force of these statistics has been so substantially undermined that they cannot sustain a finding of purposeful discrimination in salaries ...; that the defendants have not only “articulated” plausible reasons for the seeming salary disparities, but have satisfied the court of the validity of their explanations____ It follows that plaintiffs have failed to establish by a preponderance of the evidence that the Extension Service has discriminated against black employees in the matter of salaries. Id. at 54-55 (citation and footnotes omitted). The Fourth Circuit affirmed this determination by the District Court in Bazemore. See 751 F.2d 662 (1984). The appellate court referred specifically to two flaws in the plaintiffs’ statistics as grounds on which the District Court could legitimately rely in ruling for the defendant. “In the first place,” the Fourth Circuit stated, the plaintiffs’ statistics “contained salary figures which reflect the effect of pre-Act discrimination, a consideration not actionable under Title VII but permissible [only] to show the general background of the case, or intent, or to support an inference that such discrimination continued.” 751 F.2d at 672 (footnote omitted). Second, the appellate court noted that plaintiffs’ statistical study of salaries did not take into account “across-the-board and percentage pay increases which varied from county to county.” Id. The court stated that “[t]he across-the-board and percentage pay increases granted by the various counties in varying amounts, as well as simply paying higher salaries, are bound to have an effect on the salaries of the agents in the various counties.” Thus, the appellate court held that “the district court was not required to accept [the plaintiffs’ statistics] as proof [of discrimination] by a preponderance of the evidence.” Id. The court went on to say that “appropriate” statistics “should ... include all measurable variables thought to have an effect on salary level.” Id. The Supreme Court reversed. In a unanimous opinion for the Court, Justice Brennan responded to the Fourth Circuit’s “plainly incorrect” approach to statistical evidence: Importantly, it is clear that a [statistical] analysis that includes less than “all measurable variables” may serve to prove a plaintiff’s case. A plaintiff in a Title VII suit need not prove discrimination with scientific certainty; rather his or her burden is to prove discrimination by a preponderance of the evidence. 106 S.Ct. at 3009. Thus, imperfections in the data on which the analysis depends, or the omission of possible explanatory factors from a plaintiff’s statistical study, is not necessarily fatal to an inference of discrimination. “While the omission of variables from a regression analysis may render the analysis less probative than it otherwise might be,” the Justices held, “as long as the court may fairly conclude, in light of all the evidence, that it is more likely than not that impermissible discrimination exists, the plaintiff is entitled to prevail.” Id. Elsewhere in the opinion, Justice Brennan makes plain that the determination by the District Court whether discrimination exists or not “is subject to the clearly erroneous standard of appellate review.” Id. at 3008. While the Supreme Court remanded the case to the Fourth Circuit to definitely determine whether “based on the entire evidence in the record,” the District Court’s decision had been clearly erroneous, the Justices did declare, “we think that consideration of the evidence makes a strong case for finding the District Court clearly erroneous.” Id. at 3010-11 (footnote omitted). Rather than viewing the inclusion of “pre-Act” salaries in the statistical study as rendering the study fatally flawed, the Supreme Court stated that “evidence of pre-Act discrimination is quite probative.” 106 S.Ct. at 3010 n. 13. Similarly, the Supreme Court rejected the assumption made by both the District Court and the Fourth Circuit that county-to-county variations in certain pay increases undermined plaintiffs’ statistical conclusions: “Absent a disproportionate concentration of blacks in such counties, it is difficult, if not impossible, to understand how the fact that some counties contribute less to salaries than others could explain disparities between black and white salaries.” Id. at 3010. Thus, Bazemore instructs lower courts to be cautious about dismissing plaintiffs’ statistical studies as not probative simply because defendant offers some nondiscriminatory explanation for the disparities shown. Implicit in the Bazemore holding is the principle that a mere conjecture or assertion on the defendant’s part that some missing factor would explain the existing disparities between men and women generally cannot defeat the inference of discrimination created by plaintiffs’ statistics. To be sure, as the Supreme Court acknowledged in Bazemore, there may be a few instances in which the relevance of a factor to the selection process is so obvious that the defendants, by merely pointing out its omission, can defeat the inference of discrimination created by the plaintiffs’ statistics. See 106 S.Ct. at 3009 n. 10. The logic of Bazemore, however, dictates that in most cases a defendant cannot rebut statistical evidence by mere conjectures or assertions, without introducing evidence to support the contention that the missing factor can explain the disparities as a product of a legitimate, nondiscriminatory selection criterion. This court, even before Bazemore, had explicitly endorsed the same principle, most recently in a situation where the government attempted to rebut the inference of discrimination arising from evidence that blacks in the Drug Enforcement Agency were paid less and promoted less rapidly than whites. The government argued that blacks were less likely than whites to have an extra year of “specialized experience” over and above minimal qualifications. We rejected the argument because the DEA failed to introduce any evidence to substantiate its assertion: Since DEA has presented no admissible evidence that black agents are more likely than white agents to lack a second year of requisite experience, plaintiffs’ failure to account for this variable does not dilute the force of their statistical analysis; ... absent any reason to conclude that the. omitted factor correlates with race, the omission of this variable will not affect the validity of the race coefficient in the plaintiffs’ regression analysis. Segar, 738 F.2d at 1277. We think the lessons of both Bazemore and Segar apply to this case. IV. A Review op the Disparate Treatment Claims in This Case Having discussed the applicable legal principles, we now address the specific disparate treatment claims at issue in this case. Supreme Court precedent has made plain the appropriate standard for reviewing a district court’s determination that employment decisions were not the product of an unlawful discriminatory animus. We can reverse this factual finding only if it is clearly erroneous in light of all the evidence in the record or if it rests on legal error. See Bazemore v. Friday, — U.S. -, 106 S.Ct. 3000, 92 L.Ed.2d 315 (1986); Anderson v. City of Bessemer City, 470 U.S. 564, 105 S.Ct. 1504, 84 L.Ed.2d 518 (1985); Pullman-Standard v. Swint, 456 U.S. 273, 102 S.Ct. 1781, 72 L.Ed.2d 66 (1982). A. Promotions and Evaluations The Secretary of State argues that appellants’ claim of “class-wide promotion discrimination lie[s] at the heart of this case.” Appellee’s Brief at 58. We agree. Appellants claim that the State Department discriminated against women in promoting FSOs from class 5 to class 4 from 1976 to 1983. According to the government’s own evidence, fewer women than expected were actually promoted to class 4 during that time period, given the number of promotion-eligible women in class 5. The government’s own statistical analysis, whose methodology the District Court found to be more accurate than appellants’, concluded that the discrepancy between the actual and expected number of women promoted measured 1.76 standard deviations. See 616 F.Supp. at 1557; Defendant's Exhibit 8A at 14 (Table 1, Model 2). As the District Court noted, this measurement means that the probability of an underpro-motion of women this large or larger (a one-tailed inquiry) occurring randomly measures slightly less than 4%. 616 F.Supp. at 1557. As we have discussed, under a one-tailed test this number meets the 5% level set forth in Segar. But the corresponding probability of a random deviation from the expected number of women, either favoring or disfavoring women (a two-tailed inquiry), with a magnitude this large or larger is slightly less than 8%. See Defendant’s Exhibit 8A at 14 (Table 1, Model 2). Thus under a two-tailed test, this number fails to meet the 5% level. For the reasons set forth in Part III. A., we do not think this evidence alone is sufficient to prove an intent to discriminate against women. Appellants at trial, however, relied on additional evidence to prove a discriminatory motive. Appellants first point to evidence in the record of a general prejudicial attitude against women within the Foreign Service during this time period and argue that this evidence supports the proposition that the discrepancy between the actual and expected number of women promoted to class 4 results from a prejudicial attitude against women that violates Title VII. This evidence includes statements made upon cross-examination by the defense witness, Benjamin Reid, who was Undersecretary of State for Management from 1977-1981. Reid testified that the Foreign Service, as a result of traditionally being “white, male, and Ivy League,” had “set ways of doing things” and that although during his tenure the Foreign Service “had come a long way,” it nevertheless “still had a long way to go” at the time he left in correcting these biased attitudes. Tr. at 3279-80. Similarly, the appellants introduced into evidence a report written in 1977 by a committee within the State Department asserting that “both attitudinal resistance to equal employment opportunity and discriminatory behavior are still widespread in the Department.” Plaintiffs’ Exhibit 29 at 6. The appellants also introduced into evidence a report published in 1984 by the Women’s Research and Education Institute of the Congressional Caucus for Women’s Issues, which stated that “ ‘what some identify as traditional elitist attitudes have [worked] to limit severely employment opportunities for women and minorities [in the Foreign Service].’ ” Plaintiffs’ Exhibit 88 at 10 (quoting a 1981 report prepared by the U.S. Commission on Civil Rights). More specifically, as proof that the un-derpromotion of women FSOs from class 5 to class 4 resulted from a prejudicial attitude against women, the appellants relied upon evidence that the State Department believed that women FSOs had less potential for advancement than men FSOs even though men and women FSOs performed their duties with the same skill. A random sample of the evaluation reports for over 400 FSOs in classes 5 and 6 revealed that although “there was no significant difference in the performance ratings of men and women, ... the disparity between men and women [in their potential ratings] measured 2.49 standard deviations.” 616 F.Supp. at 1549 (1162) (emphasis added). As the District Court noted, this measurement means the likelihood of women being randomly underrated to this degree or greater (a one-tailed inquiry) is only about 7 times in 1,000. Id. Correspondingly, the likelihood of women randomly being either underrated or overrated to this degree or greater is 14 times in 1,000. Either way the odds are very small indeed. The relevance of this evidence to whether the underpromotion of women from class 5 to class 4 resulted from a discriminatory attitude against women is obvious. As the State Department itself asserted and the District Court expressly found, competitive promotion decisions in the Foreign Service were based primarily on an “assessment of the officer’s potential to perform at the next higher level.” 616 F.Supp. at 1555 (A 114); Defendant’s Post-Trial Brief at 92. If a biased attitude towards women was causing the State Department to underrate the potential of class 5 women FSOs in their evaluation reports, even though these women were on average performing equally as well as their male counterparts, one might well expect that this same biased attitude would be at work in the promotion decision itself. The District Court, however, never considered the evidence of a discriminatory attitude about the potential of women derived from the evaluations in deciding whether appellants had proved, by a preponderance of all the evidence, discriminatory intent in the decisions pertaining to promotions from class 5 to class 4. Rather, the District Court offered the following grounds for rejecting the evidence relating to the evaluation reports: In view of the finding that female FSO’s are promoted equally with male and given the same job opportunities, the Court finds that plaintiffs’ analysis of the disparity on potential ratings does not establish that the [evaluation reports] of female FSO’s are discriminatory in any fashion. 616 F.Supp. at 1560 (1Í 25). In our view this reasoning puts the cart before the horse. The District Court cannot determine that the State Department did not discriminate against women in promotions from class 5 to class 4 until it considers whether or not all the evidence demonstrates a biased attitude towards women and their capabilities. It cannot reject relevant evidence of discriminatory intent on the basis of a conclusion that no discrimination occurred without referen