WPCo  Bq(Ev/20/U m;٨ I8~ZLR鴆5Z#XE[yDZ2lbf+ml*]8\(zamCM Xmi"SԫIA%Hn 2&y˯sZ4?npRv9I1@)+J& WTsYZ_a*Y$)0^M^&~,RЬgcFpoi Ę q`+VccPL۝ЙS|dn m >-\#[ȋux;$mߠV]{TWS3T?w=TF1"|o(@[<Jӹ}) ?WoȈc>Aؤ@y:BV.̅ՓDfFb:͵Op=KKfV ,^Nlł#&PUNR %=S 0 :CSUN}S 1BuS 72@TU>rT^ TwT 0PT4U$U3U m5UU:LU 72UNU 0]U 0DVbV 72V B*W 72MU *U@ 72N' 1) 0U 0l 0q 0 0% 09 0= 0[ 0y 0c 0i 0._ C  0 0 09 0 0 0 0 0 0 0( 09 0G 0U 0 0 0 0  0  0  0S) 0S|| 0P 0P 0>o 0; 0; 08# 08[ 0B 0S 0S( 0P{ 0P 0 0> C9 0;S 0> 0: 0& 0, 0>  0>H 0  00Z  00  00  0- 0- 0-D 0-q 0* 0> 0 3 039 06l 06 0 0 72V  0  72J# 0|# 0'>& 0e' 0!* 09+ 0- 00 0R3 06 0-8 0-9 0'; 0'9< 0'`= 0!> 0!? 0!@ 0!A 0! C 0-,D 0'YE 0!F 0G 0H 0I 0J 0K 06M 0-FN 0-sO 0'P 0'Q 0'R 0'T 0! 0A 0vD 05G 0I 0L 0DO 0'Q 0'S 0!ET 0!fU 0!V 0!W 0X 0Y 0Z 0\ 0^ 0a 0wd 00g 0i 0l 0io 0*r 06t 0u 0x 0{ 0[~ 0 0ރ 0 0M 0 0 0'o 0 0X 0+ 0 0 0v 05 0 0 0D 06 0- 0 0´ 0 0C 0  0̿ 0| 0= 0 0 0-A 0-n 0' 0' 0' 0' 0!7 0!X 0!y 0 06M 0' 0' 0' 0' 0' 0'F 0!m 0! 0! 06 06 0< 0 06 0' 0 0 0 0\ 0 0 0 0N 0 0 0  0  0] 0% 0 0 0g 0 0 06v" 0# 0_& 0-) 0-?* 0'l+ 0', 0'- 0'. 0!0 0!)1 0!J2 06k3 0'4 0'5 0!6 0!8 0!19 0R: 0m; 0< 0= 06> 0? 06B 06C 0!E 04F 0OG 0gH 0|I 0J 0 K 0 L 0M 0!QN 0rO 0P 0Q 0R 0S 0 T 0 U 0V 0'W 0X 0i[ 0^ 06` 0!b 032c 00ed 0-e 0*f 0'g 0!i 04j 0:k 0Ul 0o 0q 0t 0kw 05zU,|Ut2 } 0R}Uu6U48 0l 0-U,˅N 0N iN 0  0ދ i 1Ŏ 0L 1 AS 0ߒ 0 1h i 0 i iYݛ 06 0 i= iҡ 0 i5 iK 03 0 i#Ϊ iX 0I i. iD0 0t 0> i\ imY 0ƴ iNv i7ķ 0 0 C Z iMt 06 0 i, iH 0- 0 i; i! 0$ iW iY4 0 0W ii iJ 0 iPy i~ 0G 0 i[ i 0 ia: i^ 0- 0-& 0'S 0'z 0' 0! 0! 0!  0!+ 0'L 0's 0' 0' 0' 0' 0!6 0!W 0!x 06 0' 0 i` C  0'# 0'J 0!q 0! 0! 0! 0 ib 7 r C~ iX 0U iE B 7 B 7 C: D5T 1e B* D- D5E iXz CN 1 Bu 1 1 1 1' 1 15 72 1 1u  1  0  0_  1;  1  1I N  1  1qN 0g 1' 1N=M? 1C 1 1Q 1 1_NN 72 1 1* 1 1wM 1N 1NNM 1 1 1( 1 16 1 72D 1v 1 1 1# 1 11 1 1? 72 1 1 M! 1 ! 1! 1" 0e"M# 1# 1# 1$ 1$ 72$%(V% 72~% 1% 72O& 1& 0' 72) 728)Mj) 1n) 72)L'*Ns* 1u* 0D*M@+MD+ 1H+ 1+MV,MZ,M^,Mb,Mf,Mj,Mn,Mr,Mv,Mz,M~,M,M,M,M,M,M,M,M,M,M,M,M,M,M,(, 72, 1- 1- 1". 1. 0"0/ 0JR0 01 0z2NX3 72Z3M3M3N3M3 0f3 724M24N64 084 C:4 725 CN5Nh5 Cj5M5 C5 C5(5 725 C6M06N46M66M:6N>6 1[@6;6 7M37 0 77W8M9 B+!9(L9Nt9Mv9(z9M9N9M9M9M9 A9M3:M7:N;:M=:MA:ME:MI:MM:MQ:(U: 72}: 72:M:M: 72:M;N; 1s!; i8;N;M;M; 0V;(,< 72T< 72< 72<N< 72< 72=;P=(= 72= 72)>N[>N]>N_> 72a>N>N>N>M>N>N> 0> 72S? 0?MC@NG@NI@NK@MM@ 72Q@ DE@ 72@M@ 0@MA 0>A 0A 72B((C 72PC 72C 72C 72C 0D 0D 0E 0FMG 0G 0H 0@I 02J 0DJ 0:@K 72zK 0 K 0L 08N 0>P 0DR 06S 0HS 0:PT 0^T 0<TL$UUNpU<U 72VN,W 72.W 72`WNW 72WNW 72WNW 72WN.X(0XMXX 0D\X 0mXN YMY 72YMEY 1IYMYNYNY 72Y 72 ZNZ(BZMjZ 72nZ BZMZ AYZ 0D[ 1u`[N[ 0F[ 0\ AM\\ 0] 16^ 72^ 0L^ 07__ 0~`M`(a 72+a 72]aNaMaNa(a 72a;a/&pbNb 1b('cNOc 72QcMc(c 72cNcMc 72cNd 1d C,d C7d )eN.e 10eNe 72e CKe CE6fN{fN}f 0-f 0g 0h 0k 0Un 0q 0s 0v 0/y 0{ 0~ 0V 0 0ˆ 0 0H 0 0 0E 0 0ԙ 0 0K 0 0 0i 0 0ެ 0 0S 0 0з 0z 05 0 0 0t 0' 0 0 0N 0  7  7  7  1U 12 7  72N 72 72+ 72]nNN 72 72 1 1 0   7  7  7 , 728Nj 72l 0 72p 72 0 0z 0X 726 72h 0Nx 0z 0X 728 72j 72 72 72 722 72d 72 72 72N, 0z. 72 72 72  72> 0zp 72 72 72N 72 72 72 72 0H 0B 72" 72T 72 72 72 72 72N 72 72 0 0 0 0 0 0  0 0 0 $ 0 0 0 < 0 H 0 T 0 ` 0 l 0x 0 0 0 0 0 0 0 0 0 0 0 0  0  0  0  0  0 0 0 0 0 0 0 0 0 0 0 0 0  0 0 0 0 0  0$! 0(" 0,# 00$ 04% 08& 0<' 0@( 0D) 0H* 0L+ 0P, 0T- 0X. 0\/ 0`0 0d1 0h2 0l3 0p4 0t5 0x6 0|7 08 09 0: 0; 0< 0= 0> 0? 0@ 0A 0B 0C 0D 0E 0F 0G 0H 0I 0J 0K 0L 0M 0pN HPnYYPNS 72SUBS 72(TMZT 0^TNvU 0$xU 7 V 1V 7 3W 72?W 7 qW 0}W 7 ]X 72iXNX 7 X 7 X 72X 7 X 7 XNX(YM)Y 72-Y BC_Y5YNZZZZZ 72qkqk 72k 72k 72ll 729l 72kl 72ll 72lllll 72m 723mNemem 72gm 72mm 72m 72mm 72/n 72anan 72n 72n 72n 72)o)o 72[o 72o\\CENSUS\PRED-1002-2-HP4100-1A.PRED.DECENNIAL.HQ.CENSUS0(9 Z6Times New Roman RegularX($USUS.,(9 Z 6Times New Roman Regular3#37=CIQYag1.a.i.(1)(a)(i)1)a)i)\  `&Times New Romanogq1 ((i$23  0  +}1h0.\0Backup3|xU,  AZ"Arial Regulari)(7$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#0(#(#23  0p(#(#('$  2  " i)&0 d d*i)(Ӎ$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#0(#(#0p(#(#23  0p(#p(#(.(3($ !USUS.,      0  i)(.3$ !USUS.,   #$%&'(hۈCEIMQUY]aAutoList1""""""""i)1)a)*(<$0  23  0` (#(#(P4n$0  0` (#(#23  0 ` (#` (#(d$0  0` (#(#0 ` (#` (#23  0 (# (#(x$0  0` (#(#0 ` (#` (#0 (# (#23  0h(#(#(Ñ$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#23  0h(#h(#(d$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#23  0(#(# #$%&'(hDCEIMQUY]aAutoList5i)1)a) #$%&'(xGaeimquy}Bullet ListBullets List64Hyperlink     #$%&'(hDCEIMQUY]aAutoList6i)1)a)     ($USUS.,    3    _Ԁ#XXThenumberofpersonpanelsequalsthenumberofpersonrecordsassociatedwithareturn.#XzX#f#  ($USUS.,    4    _Ԁ#XXThenumberofrosterentriesequalsthenumberofnameslistedinthequestionnaireroster.#XzX#f#>$"Small Circle"0  )Hairline di)*+ (_2623  ..*G+G (_25   /%` ` hp x /23  ../%` ` hp x /   *8+8 (_24  ," hp x ,23  ..," hp x ,   #$%&'(x?[_cgkoswSquaresSquare Bullet*5+5 (_23 ` ) hp x )23  ..) hp x ) ` *2+2 (_22  &hhp x &23  ..&hhp x &  */+/ (_21  #p x #23  ..#p x #  *,+, (_20 h  p x 23  .. p x h *)+) (_19  pp x 23  ..pp x   *&+& (_18   x 23  .. x   *AA (_17  5+ ` hp x 523  Ԁ5+ ` hp x 5  *GG (_16   /%` ` hp x /23  Ԁ/%` ` hp x /   *88 (_15  ," hp x ,23  Ԁ," hp x ,  *55 (_14 ` ) hp x )23  Ԁ) hp x ) ` *22 (_13  &hhp x &23  Ԁ&hhp x &  *// (_12  #p x #23  Ԁ#p x #  *,, (_11 h  p x 23  Ԁ p x h *)) (_10  pp x 23  Ԁpp x   (&& &_9   x 23  Ԁ x   (AA &_8  5+ ` hp x 523  5+ ` hp x 5  (GG &_7   /%` ` hp x /23  /%` ` hp x /   (88 &_6  ," hp x ,23  ," hp x ,  (55 &_5 ` ) hp x )23  ) hp x ) ` (22 &_4  &hhp x &23  &hhp x &  (// &_3  #p x #23  #p x #  (,, &_2 h  p x 23   p x h ()) &_1  pp x 23  pp x   &&& $_   x 23   x   2P+M 0_level1  , 8.4 <DL!823  ..  5+ 4 <DL!5  2M+M 0_level2  X 5+4 44 <DL!523  ..  5+ 4 <DL!5   d2M+M 0_level3   5+` ` ` <DL!523  ..  5+ 4 <DL!5  2J+M 0_level4   2( <DL!223  ..  5+ 4 <DL!5  2J+M 0_level5   2( <DL!223  ..  5+ 4 <DL!5  2G+M 0_level6   /% <DL!/23  ..  5+ 4 <DL!5  2G+M 0_level7  4 /% <DL!/23  ..  5+ 4 <DL!5  2D+M 0_level8  ` ,"< <<DL!,23  ..  5+ 4 <DL!5  2D+M 0_level9   ,"h hhDL!,23  ..  5+ 4 <DL!5  *2PM 0_levsl1  , 8.4 <DL!823  Ԁ  5+ 4 <DL!5  2MM 0_levsl2  X 5+4 44 <DL!523  Ԁ  5+ 4 <DL!5  2MM 0_levsl3   5+` ` ` <DL!523  Ԁ  5+ 4 <DL!5  i)2JM 0_levsl4   2( <DL!223  Ԁ  5+ 4 <DL!5  2JM 0_levsl5   2( <DL!223  Ԁ  5+ 4 <DL!5  2GM 0_levsl6   /% <DL!/23  Ԁ  5+ 4 <DL!5  (O$2GM 0_levsl7  4 /% <DL!/23  Ԁ  5+ 4 <DL!5  2DM 0_levsl8  ` ,"< <<DL!,23  Ԁ  5+ 4 <DL!5  2DM 0_levsl9   ,"h hhDL!,23  Ԁ  5+ 4 <DL!5  2PM 0_levnl1  , 8.4 <DL!823   5+ 4 <DL!5  i)2MM 0_levnl2  X 5+4 44 <DL!523   5+ 4 <DL!5  *2MM 0_levnl3   5+` ` ` <DL!523   5+ 4 <DL!5  2JM 0_levnl4   2( <DL!223   5+ 4 <DL!5  2JM 0_levnl5   2( <DL!223   5+ 4 <DL!5  2GM 0_levnl6   /% <DL!/23   5+ 4 <DL!5  2GM 0_levnl7  4 /% <DL!/23   5+ 4 <DL!5  2DM 0_levnl8  ` ,"< <<DL!,23   5+ 4 <DL!5  2DM 0_levnl9   ,"h hhDL!,23   5+ 4 <DL!5  6k o4Heading 1XXX?%2A`Arial?   XXXS\  `&Times New RomanS6 4Heading 2   6= 04Heading 3         XXX6) 4Heading 4   XXX 65 (4Heading 5     XXX  6W D4Heading 6  ;14 <DL!;    5+ 4 <DL!5 6? 24Heading 7  L     XXX6& 4Heading 8   XXX6) 4Heading 9&&&   XXX <:Default Para2AA 0Level 1 X 5+4 44 <DL!5  5+ 4 <DL!50> .Footer !>44 <DL!>:8Page Number6 4Body Text<MM :Body Text In |  5+` ` ` <DL!5  5+ 4 <DL!5 | <;M :Body Text I2  h #DDDL!#  5+ 4 <DL!5  : 8Body Text 2 XXX 2((0Caption   XXX  < :Footnote TexXXX<:Footnote Ref #$%&'(h#KEGKOSW[_cAutoList40)1)a)0> .Header !>44 <DL!>, *xl24 S tS&&&7(X72( 4 <DL27(X7XXX2( 4 <DL2 tSVVVV'dxd'dxd, *xl25 S tS&&&7(X72( 4 <DL27(X7XXX2( 4 <DL2 tSVVV)1 dxd, *xl26 S tS&&&7(X72( 4 <DL27(X7XXX2( 4 <DL2 tSVV)2 dxd, *xl27 S tS&&&7(X72( 4 <DL27(X7XXX2( 4 <DL2 tS, *xl28 S tS&&&7(X72( 4 <DL27(X7XXX2( 4 <DL2 tS, *xl29 S tS&&& 7(X7  2( 4 <DL27(X7  XXX 2( 4 <DL2 tS, *xl30 S tS&&&7(X7  2( 4 <DL27(X7  XXX2( 4 <DL2 tS, *xl31 S tS&&&7(X7  2( 4 <DL27(X7  XXX2( 4 <DL2 tS,m` *xl32 S tS&&&7(X77(X7XXX tSV=Thin Bottomdxd,xk *xl33 S tS&&&7(X7  7(X7  XXX tSVVV)3 dxd, *xl34 S tS&&&7(X7  2( 4 <DL27(X7  XXX2( 4 <DL2 tSVVV)4dxd,xk *xl35 S tS&&&7(X7  7(X7  XXX tSVVE"Thin Top/Bottom dxd<JM :Body Text I1 P  2( <DL!2  5+ 4 <DL!5 P <:FollowedHype    &Draw Object <<=8C HKKKK Level 1 Level 2 Level 3 Level 4 Level 5(-2$ !USUS.,  ($A<< c*i)TUWXYZ[\ 359=AEIMQ1- o o] TUWXYZ[\ 359=AEIMQ2 o o o] i) #$%&'(xeCcgkosw{TrianglesTriangle Bullet 2EGMS]ciouAutoList221.1.1.1 1.1.1.1.1.(UEGMSY_ekqAutoList241.1.1.1.1.1.1.1.(eEGMSY_ekqAutoList251.1.1.1.1.1.1.1.i)i)i)*(hCEKQW]cioAutoList81.1.1.1.1.1.1.1. I&mage <=8C HKKKKi)8! ,toc 1toc 1            (:3]$ !USUS.,   d ($0 ($0 0 (($0 0 0 (2$0 0 0 0 (<$0 0 0 0 0 (EhCEKQW]cioAutoList21.1.1.1.1.1.1.1.Level 1Level 2Level 3Level 4Level 5 #$%&'(hCKEGKOSW[_cAutoList36)1)a)<=:SLFl Lft Sg    &&&  XXX  (CEKQW]cioAutoList31)1)1)1)1)1)1)1) 1.1.Level 1Level 2Level 3Level 4Level 5Level 1Level 2Level 3Level 4Level 5(.3$ !USUS.,  (CEKQW]cioAutoList41)1)1)1)1)1)1)1) Level 1 Level 2 Level 3 Level 4 Level 5($$   1  (.3$ !USUS.,  E<<C&OLE 2.0 Box <=8C HKKKKi)8ll6Body Text &&&S\  `&Times New RomanSS\  `&Times New RomanS8z6Heading 1   XXXS\  `&Times New RomanS     S\  `&Times New RomanS 'dxd  d'dxdP Pd'dxd d'dxd d' dxd)!dxdx'dxd&P d d\  `&Times New Roman%2A`Arial(I Z(Times New Roman i)(EGMSY_ekqAutoList531.1.1.1.1.1.1.1.*+ (_2723  ..<:Body Text 21     A7^X2` 2 22[2h222p2}22 2x2 AXXX    A7^X2` 2 22[2h222p2}22 2x2 A  .ss,Title     5+ ` hp x 5      5+ ` hp x 5  86Plain Text   5+ ` hp x 5K<6X9`(Courier NewKS\  `&Times New RomanS  5+ ` hp x 5  8kq6Block Text    /%` ` hp x /   5+ ` hp x 5  :ee8Body Text 3   5+ ` hp x 5  5+ ` hp x 5  <qq:Body Text F1    5+ ` hp x 5  5+ ` hp x 5   <qq:Body Text Fi   , 5+ ` hp x 5  5+ ` hp x 5   2Se0Closing   #p x #  5+ ` hp x 5  <ee:annotation t   5+ ` hp x 5  5+ ` hp x 5  ,ee*Date   5+ ` hp x 5  5+ ` hp x 5  <:Document Map  C4$X ddC  5+ ` hp x 5  A $TahomaAS\  `&Times New RomanS7(X7    5+ ` hp x 5  dd<ee:endnote text   5+ ` hp x 5  5+ ` hp x 5  <:envelope add  ` ) hp x )XXX?%2A`Arial?S\  `&Times New RomanS  5+ ` hp x 5  <:envelope ret   5+ ` hp x 5?%2A`Arial?S\  `&Times New RomanS  5+ ` hp x 5  2ee0index 1   5+ ` hp x 5  5+ ` hp x 5  2qq0index 2  Z M 5+ ` hp x 5  5+ ` hp x 5  Z 2qq0index 3  Z  5+ ` hp x 5  5+ ` hp x 5  Z 2nq0index 4  Z  2(J J` hp x 2  5+ ` hp x 5  Z 2nq0index 5  Z A 2( ` hp x 2  5+ ` hp x 5  Z 2nq0index 6  Z  2( ` hp x 2  5+ ` hp x 5  Z 2nq0index 7  Z  2(> > ` hp x 2  5+ ` hp x 5  Z 2kq0index 8  Z 5 /% hp x /  5+ ` hp x 5  Z 2kq0index 9  Z  /% hp x /  5+ ` hp x 5  Z <:index headin   5+ ` hp x 5 ?%2A`Arial?S\  `&Times New RomanS   5+ ` hp x 5  ,ee*List   5+ ` hp x 5  5+ ` hp x 5  0nq.List 2   X 2( ` hp x 2  5+ ` hp x 5   0nq.List 3    2(4 4` hp x 2  5+ ` hp x 5   0kq.List 4    /%` ` hp x /  5+ ` hp x 5   0kq.List 5    /% hp x /  5+ ` hp x 5   :hh8List Bullet   8.` hp x 8  8.` hp x 8  <:List Bullet3   X 2( ` hp x 2  5+ ` hp x 5    <:List Bullet2    2(4 4` hp x 2  5+ ` hp x 5    <:List Bullet1    /%` ` hp x /  5+ ` hp x 5    <:List Bullet     /% hp x /  5+ ` hp x 5    <:List Contin4  , 5+ ` hp x 5  5+ ` hp x 5  <~:List Contin3  X 2( ` hp x 2  5+ ` hp x 5  <~:List Contin2   2(4 4` hp x 2  5+ ` hp x 5  <{:List Contin1   /%` ` hp x /  5+ ` hp x 5  <{:List Continu   /% hp x /  5+ ` hp x 5  :8List Number   8.` hp x 8  8.` hp x 8  <:List Number3   X 2( ` hp x 2  5+ ` hp x 5    <:List Number2    2(4 4` hp x 2  5+ ` hp x 5    <:List Number1    /%` ` hp x /  5+ ` hp x 5    <:List Number     /% hp x /  5+ ` hp x 5    .,macro   >4@` 0p x >K<6X9`(Courier NewKS\  `&Times New RomanS  >4@` 0p x >  < :Message Head  7$(X7  5+ ` hp x 5XXX?%2A`Arial?S\  `&Times New RomanS7(X7  5+ ` hp x 5  d<~:Normal Inden  X 2( ` hp x 2  5+ ` hp x 5  <:Note Heading   5+ ` hp x 5  5+ ` hp x 5  86Salutation   5+ ` hp x 5  5+ ` hp x 5  6o4Signature   #p x #  5+ ` hp x 5  42Subtitle     5+ ` hp x 5XXX?%2A`Arial?  S\  `&Times New RomanS  5+ ` hp x 5  <:table of aut   5+ ` hp x 5  5+ ` hp x 5  <:table of fig   5+ ` hp x 5  5+ ` hp x 5  :8toa heading   5+ ` hp x 5 XXX?%2A`Arial?S\  `&Times New RomanS   5+ ` hp x 5  .,toc 2   5+V V` hp x 5  5+ ` hp x 5  .,toc 3  M 5+ ` hp x 5  5+ ` hp x 5  .,toc 4   5+ ` hp x 5  5+ ` hp x 5  .~,toc 5   2(J J` hp x 2  5+ ` hp x 5  .~,toc 6  A 2( ` hp x 2  5+ ` hp x 5  .~,toc 7   2( ` hp x 2  5+ ` hp x 5  .~,toc 8   2(> > ` hp x 2  5+ ` hp x 5  .{,toc 9  5 /% hp x /  5+ ` hp x 5  <:1Bullet List   5+ ` hp x 5XXX  5+ ` hp x 5  <c+i :Outline001_1   /% hp x /23  ..  5+ ` hp x 5  <c+i :Outline002_1   /%` ` hp x /23  ..  5+ ` hp x 5  <f+i :Outline003_1   2(4 4` hp x 223  ..  5+ ` hp x 5  <f+i :Outline004_1  X 2( ` hp x 223  ..  5+ ` hp x 5  < :Outline005_1   /% hp x /A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline006_1   /%` ` hp x /A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  i)< :Outline007_1   2(4 4` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  i)< :Outline008_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  <`+` :Outline009_1  8.` hp x 823  ..8.` hp x 8  < :Outline010_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <]+] :Outline011_1  5+ ` hp x 523  ..5+ ` hp x 5  < :Outline012_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline013_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline014_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline015_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline016_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <c.` :Outline017_1  8.` hp x 8 23  ..8.` hp x 8  <c.` :Outline017_2  8.` hp x 8 23  ..8.` hp x 8  <`.] :Outline017_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline017_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline017_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline017_6  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline017_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline017_8  2(  hp x 2 23  ..2(  hp x 2  <].Z :Outline017_9  2(  hp x 2 23  ..2(  hp x 2  <]+] :Outline018_1  5+ ` hp x 523  ..5+ ` hp x 5  <]+i :Outline018_2 P ` ) hp x )23  ..  5+ ` hp x 5 P <W+i :Outline018_3 P  #p x #23  ..  5+ ` hp x 5 P <Q+i :Outline018_4 P  pp x 23  ..  5+ ` hp x 5 P <K+i :Outline018_5 P p    x 23  ..  5+ ` hp x 5 P <E+i :Outline018_6 P     23  ..  5+ ` hp x 5 P <E+i :Outline018_7 P  %%23  ..  5+ ` hp x 5 P <E+i :Outline018_8 P % 0*0*23  ..  5+ ` hp x 5 P <E+i :Outline018_9 P 0* ..23  ..  5+ ` hp x 5 P <f+i :Outline019_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <c.` :Outline020_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline020_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline020_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline020_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline020_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline020_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline020_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline020_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline020_9  2(  hp x 2 23  ..2(  hp x 2  <`+` :Outline021_1  8.` hp x 823  ..8.` hp x 8  <`.] :Outline022_1  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline022_2  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline022_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline022_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline022_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline022_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline022_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline022_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline022_9  2(  hp x 2 23  ..2(  hp x 2  < :Outline023_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <f+i :Outline024_1   2(4 4` hp x 223  ..  5+ ` hp x 5  < :Outline025_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  *< :Outline025_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline025_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5   #$%&'(hGIcgkosw{Large BulletLarge Bullet< :Outline025_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  (H$0   vList of Figures< :Outline025_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline025_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  vList of Tables< :Outline025_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline025_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline025_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <c.` :Outline026_1  8.` hp x 8 23  ..8.` hp x 8  <c.` :Outline026_2  8.` hp x 8 23  ..8.` hp x 8  <`.] :Outline026_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline026_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline026_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline026_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline026_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline026_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline026_9  2(  hp x 2 23  ..2(  hp x 2  < :Outline027_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <`+` :Outline028_1  8.` hp x 823  ..8.` hp x 8  <`+` :Outline028_2  8.` hp x 823  ..8.` hp x 8  <]+] :Outline028_3  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline028_4  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline028_5  5+ 4` hp x 523  ..5+ 4` hp x 5  <]+] :Outline028_6  5+ 4` hp x 523  ..5+ 4` hp x 5  <Z+Z :Outline028_7  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline028_8  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline028_9  2(  hp x 223  ..2(  hp x 2  <`+` :Outline029_1  8.` hp x 823  ..8.` hp x 8  <]+] :Outline030_1  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline030_2  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline030_3  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline030_4  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline030_5  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline030_6  5+ 4` hp x 523  ..5+ 4` hp x 5  <]+] :Outline030_7  5+ 4` hp x 523  ..5+ 4` hp x 5  <Z+Z :Outline030_8  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline030_9  2( ` hp x 223  ..2( ` hp x 2  < :Outline031_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <c.` :Outline032_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline032_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline032_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline032_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline032_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline032_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline032_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline032_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline032_9  2(  hp x 2 23  ..2(  hp x 2  <f+i :Outline033_1   2(4 4` hp x 223  ..  5+ ` hp x 5  <`.] :Outline034_1  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline034_2  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline034_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline034_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline034_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline034_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline034_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline034_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline034_9  2(  hp x 2 23  ..2(  hp x 2  <]+] :Outline035_1  5+ ` hp x 523  ..5+ ` hp x 5  <W+] :Outline035_2  /%` ` hp x /23  ..  5+ ` hp x 5<T+] :Outline035_3  ," hp x ,23  ..  5+ ` hp x 5<Q+] :Outline035_4 ` ) hp x )23  ..  5+ ` hp x 5<N+] :Outline035_5  &hhp x &23  ..  5+ ` hp x 5<K+] :Outline035_6  #p x #23  ..  5+ ` hp x 5<H+] :Outline035_7 h  p x 23  ..  5+ ` hp x 5<E+] :Outline035_8  pp x 23  ..  5+ ` hp x 5<+ :Outline035_923  ..< :Outline036_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <`+` :Outline037_1  8.` hp x 823  ..8.` hp x 8  <f+i :Outline038_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <f+i :Outline039_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <f+i :Outline040_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <c.` :Outline041_1  8.` hp x 8 23  ..8.` hp x 8  <c.` :Outline041_2  8.` hp x 8 23  ..8.` hp x 8  <`.] :Outline041_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline041_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline041_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline041_6  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline041_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline041_8  2(  hp x 2 23  ..2(  hp x 2  <].Z :Outline041_9  2(  hp x 2 23  ..2(  hp x 2  <c.` :Outline042_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline042_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline042_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline042_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline042_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline042_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline042_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline042_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline042_9  2(  hp x 2 23  ..2(  hp x 2  < :Outline043_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline044_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <c.` :Outline045_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline045_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline045_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline045_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline045_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline045_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline045_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline045_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline045_9  2(  hp x 2 23  ..2(  hp x 2  <f+i :Outline046_1   2(4 4` hp x 223  ..  5+ ` hp x 5  <f+i :Outline047_1   2(4 4` hp x 223  ..  5+ ` hp x 5  < :Outline048_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline049_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline049_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <`+` :Outline050_1  8.@` hp x 823  ..8.@` hp x 8  <f+i :Outline050_2 p  2( ` hp x 223  ..  5+ ` hp x 5 p <T+] :Outline050_3  ," hp x ,23  ..  5+ ` hp x 5<Q+] :Outline050_4 ` ) hp x )23  ..  5+ ` hp x 5<Z+i :Outline050_5 |  &p x &23  ..  5+ ` hp x 5 | <W+i :Outline050_6 | < #p x #23  ..  5+ ` hp x 5 | <Q+i :Outline050_7 P  pp x 23  ..  5+ ` hp x 5 P <B+] :Outline050_8 P  x 23  ..5+ ` hp x 5 P <K+i :Outline050_9 $   LLx 23  ..  5+ ` hp x 5 $ < :Outline051_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline051_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <`+` :Outline052_1  8.` hp x 823  ..8.` hp x 8  < :Outline053_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline054_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline054_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  < :Outline055_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <`+` :Outline056_1  8.` hp x 823  ..8.` hp x 8  <`+` :Outline056_2  8.` hp x 823  ..8.` hp x 8  <]+] :Outline056_3  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline056_4  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline056_5  5+ 4` hp x 523  ..5+ 4` hp x 5  <]+] :Outline056_6  5+ 4` hp x 523  ..5+ 4` hp x 5  <Z+Z :Outline056_7  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline056_8  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline056_9  2(  hp x 223  ..2(  hp x 2  < :Outline057_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline057_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <f+i :Outline058_1   2(4 4` hp x 223  ..  5+ ` hp x 5  < :Outline059_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline059_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  < :Outline060_1  8.'` hp x 8I%2AArialBlackI23  S\  `&Times New RomanSI%2AArialBlackI..S\  `&Times New RomanS8.'` hp x 8  <`+` :Outline061_1  8.` hp x 823  ..8.` hp x 8  < :Outline062_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline062_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  < :Outline063_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <f+i :Outline064_1  X 2( ` hp x 223  ..  5+ ` hp x 5  < :Outline065_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline065_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  < :Outline066_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline067_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <c.` :Outline068_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline068_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline068_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline068_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline068_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline068_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline068_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline068_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline068_9  2(  hp x 2 23  ..2(  hp x 2  < :Outline069_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <f+i :Outline070_1   2(4 4` hp x 223  ..  5+ ` hp x 5  <`.] :Outline071_1  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline071_2  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline071_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline071_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline071_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline071_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline071_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline071_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline071_9  2(  hp x 2 23  ..2(  hp x 2  <f+i :Outline072_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <f+i :Outline073_1   2(4 4` hp x 223  ..  5+ ` hp x 5  < :Outline074_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline075_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <f+i :Outline076_1   2(4 4` hp x 223  ..  5+ ` hp x 5  <`+` :Outline077_1  8.` hp x 823  ..8.` hp x 8  < :Outline078_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline078_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <W+] :Outline079_1  /%` ` hp x /23  ..  5+ ` hp x 5< :Outline080_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline080_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  <f+i :Outline081_1  X 2( ` hp x 223  ..  5+ ` hp x 5  < :Outline082_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline083_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <c.` :Outline084_1  8.@` hp x 8 23  ..8.@` hp x 8  <c.` :Outline084_2  8.@` hp x 8 23  ..8.@` hp x 8  <`.] :Outline084_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline084_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline084_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline084_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline084_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline084_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline084_9  2(  hp x 2 23  ..2(  hp x 2  <f+i :Outline085_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <`+` :Outline086_1  8.` hp x 823  ..8.` hp x 8  <`+` :Outline086_2  8.` hp x 823  ..8.` hp x 8  <]+] :Outline086_3  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline086_4  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline086_5  5+ 4` hp x 523  ..5+ 4` hp x 5  <Z+Z :Outline086_6  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline086_7  2( ` hp x 223  ..2( ` hp x 2  <Z+Z :Outline086_8  2(  hp x 223  ..2(  hp x 2  <Z+Z :Outline086_9  2(  hp x 223  ..2(  hp x 2  <f+i :Outline087_1  X 2( ` hp x 223  ..  5+ ` hp x 5  < :Outline088_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  <f+i :Outline089_1   2(l l` hp x 223  ..  5+ ` hp x 5  <f+i :Outline090_1   2(l l` hp x 223  ..  5+ ` hp x 5  <]+] :Outline091_1  5+ ` hp x 523  ..5+ ` hp x 5  <W+] :Outline091_2  /%` ` hp x /23  ..  5+ ` hp x 5<T+] :Outline091_3  ," hp x ,23  ..  5+ ` hp x 5<Q+] :Outline091_4 ` ) hp x )23  ..  5+ ` hp x 5<N+] :Outline091_5  &hhp x &23  ..  5+ ` hp x 5<K+] :Outline091_6  #p x #23  ..  5+ ` hp x 5<H+] :Outline091_7 h  p x 23  ..  5+ ` hp x 5<E+] :Outline091_8  pp x 23  ..  5+ ` hp x 5<+ :Outline091_923  ..<]+] :Outline092_1  5+ ` hp x 523  ..5+ ` hp x 5  <W+] :Outline092_2  /%` ` hp x /23  ..  5+ ` hp x 5<T+] :Outline092_3  ," hp x ,23  ..  5+ ` hp x 5<Q+] :Outline092_4 ` ) hp x )23  ..  5+ ` hp x 5<N+] :Outline092_5  &hhp x &23  ..  5+ ` hp x 5<K+] :Outline092_6  #p x #23  ..  5+ ` hp x 5<H+] :Outline092_7 h  p x 23  ..  5+ ` hp x 5<E+] :Outline092_8  pp x 23  ..  5+ ` hp x 5<+ :Outline092_923  ..<`+` :Outline093_1  8.` hp x 823  ..8.` hp x 8  < :Outline094_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline095_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  < :Outline096_1   /% hp x /A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  <f+i :Outline097_1  X 2( ` hp x 223  ..  5+ ` hp x 5  <]+] :Outline098_1  5+ :` hp x 523  ..5+ :` hp x 5  <c+i :Outline098_2 v  /% hp x /23  ..  5+ ` hp x 5 v <`+i :Outline098_3 v : ," hp x ,23  ..  5+ ` hp x 5 v <]+i :Outline098_4 v  )B Bhp x )23  ..  5+ ` hp x 5 v <Z+i :Outline098_5 |  &p x &23  ..  5+ ` hp x 5 | <W+i :Outline098_6 | < #p x #23  ..  5+ ` hp x 5 | <Q+i :Outline098_7 P  pp x 23  ..  5+ ` hp x 5 P <B+] :Outline098_8 P  x 23  ..5+ ` hp x 5 P <K+i :Outline098_9 $   LLx 23  ..  5+ ` hp x 5 $ < :Outline099_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  (\ Y`SymbolKr`Wingdings< :Outline099_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  <6X9`(Courier New%2AArialBlack< :Outline099_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline099_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  $Tahoma< :Outline100_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  VVV>630 < :Outline100_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline100_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  V##$%&'(x nACGKOSW[_ExecSum1(i)1)a)< :Outline100_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5   #$%&'(hLEGKOSW[_cAutoList15)1)a)M -, dc< :Outline100_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline100_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  (%EGMSY_ekqAutoList211.1.1.1.1.1.1.1.< :Outline100_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  VVVIA8 < :Outline100_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline100_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  1)  V< :Outline101_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  V-%VC;8    < :Outline101_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline101_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  V# V@8C0< :Outline101_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  V$V:20 < :Outline101_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline101_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  VPH< VVVVVOGC0  < :Outline101_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  V>60V-%< :Outline101_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline101_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5   dVC;8<f+i :Outline102_1  X 2( ` hp x 223  ..  5+ ` hp x 5  < :Outline103_1  X 2( ` hp x 2A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  V$"V:2  < :Outline103_2   /%` ` hp x /K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline103_3   ," hp x ,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  V/'  V< :Outline103_4  ` ) hp x )A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  VVG?G8    VIA< :Outline103_5   &hhp x &K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline103_6   #p x #GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  5+ ` hp x 5  V$$QIK 4dd  V<4'0 < :Outline103_7  h  p x A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  5+ ` hp x 5  VV>6'0VVVVVVV^V',dd dd  dd< :Outline103_8   pp x K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  5+ ` hp x 5  < :Outline103_9   x GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS5+ ` hp x 5  VKCC$ VVVrjG<dddddd< :Outline104_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  VQIC€d€d V$$F>   <c.` :Outline105_1  8.` hp x 8 23  ..8.` hp x 8  <c.` :Outline105_2  8.` hp x 8 23  ..8.` hp x 8  <`.] :Outline105_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline105_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline105_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline105_6  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline105_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline105_8  2(  hp x 2 23  ..2(  hp x 2  <].Z :Outline105_9  2(  hp x 2 23  ..2(  hp x 2  <`.] :Outline106_1  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline106_2  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline106_3  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline106_4  5+ ` hp x 5 23  ..5+ ` hp x 5  <`.] :Outline106_5  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <`.] :Outline106_6  5+ 4` hp x 5 23  ..5+ 4` hp x 5  <].Z :Outline106_7  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline106_8  2( ` hp x 2 23  ..2( ` hp x 2  <].Z :Outline106_9  2(  hp x 2 23  ..2(  hp x 2  <f+i :Outline107_1   2(l l` hp x 223  ..  5+ ` hp x 5  <`+` :Outline108_1  8.` hp x 823  ..8.` hp x 8  < :Outline109_1  8.` hp x 8A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS8.` hp x 8  PHGdd222d<`+` :Outline110_1  8.@` hp x 823  ..8.@` hp x 8  <`+` :Outline110_2  8.@` hp x 823  ..8.@` hp x 8  <]+] :Outline110_3  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline110_4  5+ ` hp x 523  ..5+ ` hp x 5  <]+] :Outline110_5  5+ 4` hp x 523  ..5+ 4` hp x 5  <]+] :Outline110_6  5+ 4` hp x 523  ..5+ 4` hp x 5  <Z+Z :Outline110_7  2( ` hp x 223  ..2( ` hp x 2  VVVVVVF>22dF>22 (-2$ !USUS.,  VaY<3+0 d d d3+0 d d ddd)!dxdx,35;AGMSY_31.1.1.1.1.1.1.1.&p d PdV'dxd)!dxdx VVVVVVB:s0 d #$%&'(hEGKOSW[_cAutoList43)1)a)[ [d #$%&'(hlEGKOSW[_cAutoList23--------)1)a)(EGMSY_ekqAutoList111.1.1.1.1.1.1.1.(EGMSY_ekqAutoList121.1.1.1.1.1.1.1.(EGMSY_ekqAutoList131.1.1.1.1.1.1.1. #$%&'(hEGKOSW[_cAutoList44)1)a) #$%&'(h EGKOSW[_cAutoList45)1)a)P #$%&'(h\EGKOSW[_cAutoList16)1)a) #$%&'(hlEGKOSW[_cAutoList17)1)a) #$%&'(hEGKOSW[_cAutoList38"""""""")1)a)<PP@:QuickFormat5XXX  cb  XXX  cb  <PP@:QuickFormat7XXX  cb  XXX  cb  (hCEKQW]cioAutoList71.1.1.1.1.1.1.1.(hCEKQW]cioAutoList91.1.1.1.1.1.1.1.(գEGMSY_ekqAutoList101.1.1.1.1.1.1.1.(EGQ[eoyAutoList331.1 1.1 1.1 1.1 1.1 1.1 1.1 1.1  #$%&'(hKEGKOSW[_cAutoList26)1)a)a, b, c,Level 1Level 2Level 3Level 4Level 56#,4Quick a. 6  .0 6(WEGOW_gowAutoList59(1)(1)(1)(1)(1)(1)(1)(1) #$%&'(hEGKOSW[_cAutoList54"""""""")1)a)(EGMSY_ekqAutoList271.1.1.1.1.1.1.1. #$%&'(h#KEGKOSW[_cAutoList28)1)a) #$%&'(hEGKOSW[_cAutoList29--------)1)a) #$%&'(h|EGKOSW[_cAutoList30--------)1)a) #$%&'(hJEGKOSW[_cAutoList31)1)a)9" #$%&'(hEGKOSW[_cAutoList60````````)1)a) #$%&'(hKEGKOSW[_cAutoList32)1)a)1, 2, 3,Level 1Level 2Level 3Level 4Level 5 #$%&'(hKEGKOSW[_cAutoList61)1)a) #$%&'(` EGKQUY]aeAutoList58a.1)a) #$%&'(h|EGKOSW[_cAutoList18)1)a) #$%&'(hEGKOSW[_cAutoList55"""""""")1)a) #$%&'(pEGKOSW[_cAutoList39"""""""")1)a) #$%&'(hKEGKOSW[_cAutoList62)1)a) #$%&'(hEGKOSW[_cAutoList19)1)a)(EGMSY_ekqAutoList411.1.1.1.1.1.1.1. #$%&'(hKEGKOSW[_cAutoList63)1)a) #$%&'(hKEGKOSW[_cAutoList64)1)a)L" #$%&'(h<EGKOSW[_cAutoList20)1)a)(EGQ[eoyAutoList341.1 1.1 1.1 1.1 1.1 1.1 1.1 1.1  #$%&'(hKEGKOSW[_cAutoList65)1)a)(EGMSY_ekqAutoList351.1.1.1.1.1.1.1. #$%&'(hLEGKOSW[_cAutoList66)1)a) #$%&'(hLEGKOSW[_cAutoList67)1)a) #$%&'(h#LEGKOSW[_cAutoList68)1)a) #$%&'(h3LEGKOSW[_cAutoList69)1)a)1. #$%&'(hKEGKOSW[_cAutoList70)1)a) #$%&'(hKEGKOSW[_cAutoList71)1)a)(ŤEGMSY_ekqAutoList371.1.1.1.1.1.1.1. #$%&'(hCKEGKOSW[_cAutoList42)1)a) #$%&'(hLEGKOSW[_cAutoList72)1)a)4#,2Quick 1.F  .0 F #$%&'(hKEGKOSW[_cAutoList46)1)a) #$%&'(hKEGKOSW[_cAutoList47)1)a) #$%&'(hKEGKOSW[_cAutoList48)1)a) #$%&'(hKEGKOSW[_cAutoList49)1)a)1.i)(:EGQ[eoyAutoList511.1 1.1 1.1 1.1 1.1 1.1 1.1 1.1 o1.1 (EGMSY_ekqAutoList521.1.1.1.1.1.1.1.VL:head 13 bo01PRED exec sum    cb   i #$%&'((Kxx  bwxyz    cb   i #$%&'((Kxx  bwxyzq1.1. #$%&'(hKEGKOSW[_cAutoList56)1)a)vm:\office\wpwin\template\standard.wpt #$%&'(hSLEGKOSW[_cAutoList77)1)a)6b$Triangle0  #$%&'(hCLEGKOSW[_cAutoList76)1)a) #$%&'(hcLEGKOSW[_cAutoList78)1)a)i) #$%&'(`CLEGKOSW[_cAutoList82)1)a) #$%&'(hLEGKOSW[_cAutoList88)1)a) #$%&'(hLEGKOSW[_cAutoList89)1)a) #$%&'(hsLEGKOSW[_cAutoList91)1)a)<ss:C1Ctr BoldH     2( 4 <DL2 &&&     2( 4 <DL2  <:P1Stand Par      2( 4 <DL2,&&&    2( 4 <DL2   *ZZ(CM3  2( 4 <DL22( 4 <DL2  *ZZ(CM4  2( 4 <DL22( 4 <DL2  i)(W$ @*   A i)<77@n4:QuickFormat1)XXX  )XXX  6$ Diagonal Lines 1dd *(db$Large Circle0 B?21, 2, 3,NumbersF1  .0  FA, B,Level 1Level 2Level 3Level 4Level 5P1M#,A, B,Uppercase LettersX  .X0 Mi)i)i)page page chapter volume footnote endnote $%sessprattH #$%&'(xLGgkosw{Smiley FaceMr. Smiley Facei)i)Fi)8?OU[agmsyLettersLettersA.A.A.A.A.A.A.A.(Chainlinkdd3Gradient 1dd vLIST OF TABLES #$%&'(x[+A_cgkosw{DiamondsDiamond Bullet````````v`G,(Rectangular Gradientdd A&"Circular Gradientdd <wz:Normal Bulle   X /%4 4 <DL/ UK XXX  2( 4 <DL2 US    8qq6Title Page   2( 4 <DL2  UK   US   2( 4 <DL2  < :Outline001_2   ," <DL,K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_3  4 ) <DL)GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_4   &hhDL&A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_5   #DL#K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_6  <  DL GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_7   ppLA(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_8   LK<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline001_9  D    LGKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_2  t /%P P <DL/A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_3   ) <DL)GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_4  ` &<<DL&A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_5   #DL#K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_6    DL GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_7  h DDLA(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_8   LK<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline002_9   LGKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS2( 4 <DL2  < :Outline003_2   ," <DL,K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_3  4 ) <DL)GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_4   &hhDL&A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_5   #DL#K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_6  <  DL GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_7   ppLA(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_8   LK<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline003_9  D    LGKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_2   ," <DL,GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_3  4 ) <DL)GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_4   &hhDL&A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_5   #DL#K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_6  <  DL GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_7   ppLA(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_8   LK<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline004_9  D    LGKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_2   ," <DL,K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_3  4 ) <DL)GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_4   &hhDL&A(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_5   #DL#K<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_6  <  DL GKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_7   ppLA(\ Y`SymbolA23  S\  `&Times New RomanSA(\ Y`SymbolA..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_8   LK<6X9`(Courier NewK23  S\  `&Times New RomanSK<6X9`(Courier NewK..S\  `&Times New RomanS  2( 4 <DL2  < :Outline005_9  D    LGKr`WingdingsG23  S\  `&Times New RomanSGKr`WingdingsG..S\  `&Times New RomanS  2( 4 <DL2  9 359=AEIMQ4PPoPo #$%&'(h<EGKOSW[_cAutoList14)1)a)i)*i)i)\  `*Times New RomanTTC\  P6QP\  `*Times New RomanTTXXP\  P6QXP\  `*Times New RomanTT\  P6QP\  `*Times New RomanTT^\  P6QP\  `*Times New RomanTT  k\  P6Q P%2A`ArialTTomanTT&&Q2PQ&Pi)*(EEGMSY_ekqAutoList571.1.1.1.1.1.1.1.0դEGMSY_ekqAutoList501.1.1.1.1.1.1.1.2kn0Level 2   X /%4 4 <DL/XXX  2( 4 <DL2   8Pi)*<K >:Heading 1 No  2( 4 <DL2   2( 4 <DL2  i)**>>(CM2  2( 4 <DL22( 4 <DL2  *ZZ(CM1  2( 4 <DL2f&2( 4 <DL2  *ZZ(CM7  2( 4 <DL2f&2( 4 <DL2  i)**ZZ(CM8  2( 4 <DL2(2( 4 <DL2  *ZZ(CM9  2( 4 <DL2(2( 4 <DL2  ,ZZ*CM10  2( 4 <DL2(2( 4 <DL2  i)*i) A.;JRA.W<:P1-Stand ParXX!XXXX!XX!XXXX!cA.#A.<9:N7-3" Block   /%p p` hp x/S\  `&Times New RomanS   8.` hp x (#8S\  `&Times New RomanS4<=><..{V:TT-Table Tit  X(XXX(   X(XXX( 2$Square0  @?A.E3XA.ZA.1.0 ,P1.PYT 81AutoList111AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 82AutoList112AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 83AutoList113AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 84AutoList114AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 85AutoList115AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 86AutoList116AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 87AutoList117AutoList11  8.` hp x (#8        >4X` hp x (#>  PYT 88AutoList118AutoList11  8.` hp x (#8        >4X` hp x (#>  TYT :2Bullet List2Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :3Bullet List3Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :4Bullet List4Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :5Bullet List5Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :6Bullet List6Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :7Bullet List7Bullet List  8.` hp x (#8        >4X` hp x (#>  TYT :8Bullet List8Bullet List  8.` hp x (#8        >4X` hp x (#>  PYT 81AutoList101AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 82AutoList102AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 83AutoList103AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 84AutoList104AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 85AutoList105AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 86AutoList106AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 87AutoList107AutoList10  8.` hp x (#8        >4X` hp x (#>  PYT 88AutoList108AutoList10  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList91AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList92AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList93AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList94AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList95AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList96AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList97AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList98AutoList9  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList81AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList82AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList83AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList84AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList85AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList86AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList87AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList88AutoList8  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList71AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList72AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList73AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList74AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList75AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList76AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList77AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList78AutoList7  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList61AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList62AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList63AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList64AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList65AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList66AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList67AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList68AutoList6  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList51AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList52AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList53AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList54AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList55AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList56AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList57AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList58AutoList5  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList41AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList42AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList43AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList44AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList45AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList46AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList47AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList48AutoList4  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList31AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList32AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList33AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList34AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList35AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList36AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList37AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList38AutoList3  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList21AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList22AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList23AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList24AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList25AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList26AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList27AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList28AutoList2  8.` hp x (#8        >4X` hp x (#>  LYT 61AutoList11AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 62AutoList12AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 63AutoList13AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 64AutoList14AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 65AutoList15AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 66AutoList16AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 67AutoList17AutoList1  8.` hp x (#8        >4X` hp x (#>  LYT 68AutoList18AutoList1  8.` hp x (#8        >4X` hp x (#>  ^:endnote refeendnote referenceXXXWS\  `*Times New RomanTTW        WS\  `*Times New RomanTTW\  `*Times New RomanTTC\  P6QP\  `*Times New RomanTTXXP\  P6QXP\  `*Times New RomanTT^\  P6QP\  `*Times New RomanTT  k\  P6Q P  *Times New RomanTTm0Xhi)\  `*Times New RomanTTi)2ss0Style 2   8.E MU] #8-  -    8.E MU] #8  2yy0Style 3   h 2(D D4 <DL2-    2( 4 <DL2-     6P8 CSY_ekqw}Numbers 2Numbers1.1.1.1.1.1.1.1.6P*6P,ZZ*CM12  2( 4 <DL22( 4 <DL2  6Pi)6P6P*6P6Pi)? Triple+l d d d + ($USUS.,    1    _&-B%XXTheWindowRock,ArizonaLCOwasomittedfromthestudybecausesomecriticaldataforthis  oneLCOwasmissing.#XzX%&-Be#  TXEnonresponserelisting reinterviewing PSCESCAP AREX RMIE NRFUmailback DMAF CEFU$undeliverability LCOsReinterviewcaregiversmicrodatanonshelteredunduplicationunduplicate LUCAgeocodingVitranolisters deliverabilityRecanvasscoloniasgeocodedfrontloading CIFUinmoverUU TQAWHUHEVacants PSARSOURCEnonresponding subpopulations DCAR KFI RCC"idenitification IDsnonvalid RCCsreinterviewedreinterviewsrelistpoststratarosterednonmatches ROsoutmoversnonmoverspercentsinmoversnonmoversubnationalintercensalovercountFactFinder TEAsnonreseponse DSFsnowbirdscoloniapromotoras KFPnonsamplingPoststratumBonferroniunweightedmatrixedlisternonmatchedovercountedunivariatemultivariateOvercountsgeocodableNumident nonrespondentsreengineeringnonrelativeInupiaqInupiattypologiesnonrelativescoresident&multigenerationalcohabitors$operationalizing&misclassificationundercoverage PhDs DMD ERI SPs GQsMaricopaPinalLafourcheDutchess JeffersonvillemunicipiosunswappedcallbacksungeocodedrecanvassedLocatabilitycrewleadersflashcardsdatasethousematesrecoded"Primary PrimaryproactivelyscopingdiscriminanttimelineGeoTeltimelinesBaumgardnerMattinglyBerkowitzBrownriggBurchamMangarooCasparChesnutWhitworthLestina KohnredeliverynonminorityStackhouseZajac nonsubstitutednonimputedrecontactEnnisMachowskiPalumboChristensonZayatzWolter ORC CATIMcNallyRuhnkePenningtonNoncityRosenthalA.C.E.RothhaasPettaWestat LCOTenebaum Moul ImelSheppardViatorAlbertiKeathleyZelenakStemperConklinmailoutMorgansteinJohanson CAPIoutsourceSystem System$noninteroperable PAMSmodularizedhardcopy TRWStARSremailingLarwoodTrenthamnonincentiveObenski MAFSchwedehousematei)*i)*i)i)**i)*i)*i)*i)*i)*i)*i)* !USUS.,  _8XXdd8  AppendixA     SUMMARYRESULTSOFINDIVIDUALEVALUATIONSANDEXPERIMENTS  |  CategoryAResponseRatesandBehaviorAnalysis  H SummariesofthetwelveevaluationsintheResponseRatesandBehaviorAnalysiscategory  d follow.  J  EvaluationA.1.a,TelephoneQuestionnaireAssistance (Chesnut2003a)    TheTelephoneQuestionnaireAssistanceprogram(TQA)wasimplementedtoassistthepublic   incompletingtheircensusforms.Sixlanguagespecificnationaltollfreenumberswereprinted x  onCensusquestionnairesandLanguageAssistanceGuides.TheEnglishandSpanishtollfree ^  numbersconnectedtoanInteractiveVoiceResponse(IVR)systemwhereacallerobtained D  informationbyselectingfromaseriesofmenuoptions,andifneeded,wastransferredtoan *z  agent.ThetollfreenumbersfortheAsianlanguages(Chinese,Korean,Vietnamese,and ` Tagalog)connecteddirectlytobilingualagents.ThisevaluationprofiledtheCensus2000TQA F programand,whereappropriate,assessedtheperformanceoftheTQAsystem. , TheTQAprogramreceived6millioncalls,withapproximately51percentservicedbyanagent.  Therewerethreepeaksforthedailytotalcallvolume.Thefirstpeakoccurredaftertheinitial  mailoutofcensusquestionnaires,thesecondpeakoccurredafterthemailoutofthereminder t postcard,andathirdsmallerpeakoccurredtheweekofCensusDay.Eachofthepeaksoccurred Z onMondays.Excludingthepeakdays,atrendwasobservedinwhichMondaywasthehighest @ callvolumedaywithagradualdeclineincallvolumethroughouttheweekendingwithalow &v callvolumeonSunday.  \ TheIVRsystemallowedcallerstoobtainorenterinformationbyselectingfromaseriesof ( menuoptions.TheCensusBureaufoundthatcallersprimarilyusedtheIVRsystemtoobtain  informationoncompletingacensusform.Second,theyusedtheIVRsystemtorequestacensus  formbymailandthird,callersusedtheIVRsystemtoobtaingeneralinformationandother   informationpertainingtothecensus.Ofthemainservicesprovidedbyanagent,themost p!  frequentlyrequestedservicewastherequestforacensusform.Callersalsorequestedanswers V"! tofrequentlyaskedquestionsaboutthecensus,requestedanswersconcerningspecificitemson <#" thecensusquestionnaire,orregisteredacomplaintaboutthecensus. "$r# Basedontheresultsofthisevaluation,thefollowingrecommendationsweremade: %>!% 2  2 "0    PredictthecallvolumesonpeakcallvolumedaysbasedonCensus2000experience.2 ݌' #'(#(# Ќ  2  2{ "0    Monitortheperformanceofthenetworkproviderbasedontheircontractual )$) requirements.2{ ݌l*%*(#(# Ќ   8,', 2  2"0    DeliverthecensusmailingpiecesonadayotherthanMondaytoavoidanadditiveeffect  duetomailingstrategyanddayoftheweek.2݌(#(# Ќ  2  2"0    ConductfutureresearchinassessingtheexpandeduseofIVRtechnologies.2݌b(#(# Ќ  2  2"0    Provideonsitetechnicalsupporttoallcallcenters2݌. ~(#(# Ќ  2  2H"0    ProvideequallevelsofserviceinbothEnglishandSpanishsystems.2Hc݌ J(#(# Ќ  EvaluationA.1.b,TelephoneQuestionnaireAssistanceCustomerSatisfactionSurvey     (Stevens2002)    ThepurposeofthisevaluationwastomeasurehowsatisfiedcallerswerewiththeTQA x  operationforCensus2000.TheCensusBureaumeasuredcustomersatisfactionfromtwo ^  perspectives:thecallerandtheagentwhohandledthecall. D  Overall,callersweresatisfiedwiththeTQAoperation.Atleast72percentoftherespondentsto ` thecustomersatisfactionsurveyrepliedfavorably.Thesurveyquestionsaskedabouteaseof F movingthroughtheautomatedmenusystem,quicknessoftheagentinunderstandingtheir , request,agentslevelofinterestinhelping,overallsatisfactionwiththecall,andothercustomer  concerns.  Overall,agentswidelysupportedmostaspectsoftheTQAoperationmentionedontheagent t debriefingquestionnaire.Onlythreeoutofnineteenquestionswereviewednegativelybythe Z majorityoftheagents.Theagentssatisfactionwiththeoperationsupportsthecallers @ satisfaction.Theagentsagreedthat:theyunderstoodthecallersrequests,thevisualdesignof &v theOperatorSupportSystemmadeiteasytoreadthepreparedanswers,traininghelpedthem  \ understandCensusconcepts,anditwaseasyforthemtousetheOperatorSupportSystemtofind B theinformationthatcallersrequested.However,theagentsfeltthecallersseemeddissatisfied ( whentheyrepeatedthesameverbatiminformationandtheyfelttheycouldhaveusedmore  practicewiththeOperatorSupportSystembeforefieldingcalls.  Basedontheresults,recommendationsare: p!   #$%&'((CKx3" i"   ip2CK3  0    Continuetoresearchthecallersexpectationatthefirstandsubsequentmenuselections <#" intheIVRsystem,aspartoforpriortodevelopment. ip݌"$r#(#(# Ќ  ! ! #$%&'((#Kxx #$%&'(" i"   ip2#K3  0    Researchthecallersexpectationsofwaitingtimesandmakeadjustmentsaccordinglyto %>!% themaximumtimeagentsshouldkeepcallerswaiting. ip݌&$"&(#(# Ќ  " i"   i2#K3  0    DesigntheOperatorSupportSystemscriptsothatlessinformationisrepeatedwhenthe (#( agentsarerespondingtoanincomingquestion.Inaddition,increasetrainingonhowthe )$) agentscanreadtheOperatorSupportSystemscriptappropriately. i݌l*%*(#(# Ќ   8,', " i"   i2#K3  0    Providetheagentswithextrapracticetimeandincludemorerealisticexamplesof  differenttypesofcalls. i݌(#(# Ќ  " i"   i 2#K3  0    Allowagentstorespondtorequestsforreplacementformsassoonasthereminder b postcardsaredeliveredtomailout/mailbackaddresses. i !݌H(#(# Ќ  " i"   i="2#K3  0    Providetheagentsand/ortheIVRsystemwithtoolsforverifyingwhethertheCensus  d Bureaureceivedacallerscensusform,beingsuretoaddressconfidentialityissues. i="j"݌ J(#(# Ќ  EvaluationA.2.b,InternetDataCollection (Whitworth2002)    Census2000wasthefirstU.S.censustoincludeanInternetonlinereportingcapability.   Thoughitwasmetwithmanychallenges,theInternetdatacollection(IDC)wasanoperational x  success.Itprovedtobesecureandtherewerenohardwareorsoftwarefailuresandnoknown ^  securitybreaches.However,itdidnotfullyreachitspotential.Thisisprobablybecauseofa D  consciousdecisionbytheCensusBureaunottoadvertisethisresponsemode.TheIDCsystem *z  couldhavehandledtensofmillionsofforms.Instead,theCensusBureaureceived89,123initial ` onlinerequestsforanInternetcensusform.About16.7percentoftheseinitialrequestswere F invalid(mostlyrequestsforthelongform).Therewere63,053householdsrepresenting169,257 , personsthatwerecountedusingjusttheInternetcensusform.Fewerthan4percentofInternet  submissionshadotherreturnsfromdifferentmodes.  TheInternetformcollecteddataforsixpersons,withacontinuationrostersimilartotheshort t formpaperquestionnaire.Thisrestrictionofasixpersonformrequiredfollowupforlarge Z households,butdidnotgreatlyaffectCoverageEditFollowup(CEFU)operations.Fewerthan2 @ percentoftheInternethouseholdsreportedhouseholdsizetobelargerthansix.Internet &v householdshadapproximatelythesamereportedhouseholdsizeasreportedforallresponse  \ modes.ThereweresomedifferencesbetweenthedemographicmakeupfortheInternet B householdmemberswhencomparedtotheoverallpopulation.Thisisnotsurprisingwhenone ( considersthetypicalInternetuser.Demographichighlightsinclude:  " i"   i+2#K3  0    Therewereslightlymoremales(52.0percent)thanfemales(48.0percent). i++݌ (#(# Ќ  " i"   i,2#K3  0    TherewerehigherpercentagesofWhiteandnonHispanichouseholdmembersthanfor V"! allresponsemodes. i,,݌<#"(#(# Ќ  " i"   i.2#K3  0    Therewasahigherpercentageofpersonsbetweentheagesof2554respondingonthe %X $ Internet,ascomparedtoallresponsemodes. i./.݌%>!%(#(# Ќ  TheInternetisheretostay.TheexactformandfunctionofCensusInternetoptions,however,is ' #' largelyundeveloped.Recommendationsinclude: (#( " i"   i 02#K3  0    Focusfutureresearchnotonlyonhowtosecurelyimplementtheformitself,butalsoon l*%* howtopromulgatetheInternetformasamajorresponseoption. i 060݌R+&+(#(# Ќ   8,', " i "   i12#K3  0    FocusfutureresearchonhowtouseInternetresponseasatooltoincreasedataqualityby  implementingrealtimedatafeedbackandanalysis. i11݌(#(# Ќ  EvaluationA.2.c,Census2000InternetWebSiteandQuestionnaireCustomerSatisfaction b Surveys (Stapleton2002) H Aspartofacomprehensiveplantosimplifypublicparticipationandtoincreaseresponseratesto  d Census2000,theCensusBureaudesignedasinglewebsitetoserviceInternetusers.Thesite  J containedtwomajorcomponents:InternetQuestionnaireAssistance(IQA)andIDC.Customer  0 satisfactionsurveyswereconductedandthesurveyresultswereusedtoanalyzethedegreeof    respondentsatisfactionwitheachsystem.    MostrespondentswerenotsatisfiedwiththeIQA.Nearly62percentoftherespondents x  indicatedthat,overall,theywerenotatallsatisfiedwiththeInternethelpscreens.Whilenearly ^  77percentoftherespondentsfounditeasyorveryeasytounderstandthehelpscreen D  information,about58percentsaiditwasnotatalleasytofindthehelptopicsforwhichthey *z  weresearchingand65percentstatedthatthehelpscreeninformationwasnotatallhelpful. ` Thesefindingssuggestthatwhiletheinformationpresentedonthesitewaseasytointerpret,it F maynothavebeentheappropriateinformationfortheusers.Thoserespondentswhodidfind , theinformationhelpfulweremoresatisfiedoverall.WhiletheinformationonIQAwaseasyto  understand,itwasdifficulttolocateandgenerallyunhelpful.Inshort,theIQAdidnotprovide  theinformationthatrespondentswereseeking.However,thehighcorrelationbetween  helpfulnessandoverallsatisfactionindicatesthattheCensusBureaumightimprovecustomer t satisfactionbyfocusingonIQAelementsthatarehelpfultousers. Z TheCensusBureaumeasuredsatisfactiononsevenaspectsoftheCensus2000InternetForm: &v timerequiredtoloadtheform,movingthroughtheform,availabilityofhelpscreens,  \ understandingthehelpscreeninformation,easeofsendingtheform,securityandconfidentiality B procedures,andoverallsatisfaction.Respondentswerelargelysatisfiedwithmostoftheseven ( aspects.Thepercentofrespondentsindicatingtheyweresatisfiedorverysatisfiedwitha  specificaspectwasashighas94percent(fortheitemeaseofsendingform).However,  satisfactionlapsedslightlyforthetwoitemswhichdealtwithhelpscreens:availabilityofhelp   screensandunderstandingthehelpinformation(74percentand73percent,respectively). p!  Overall,91percentofrespondentsweresatisfiedwiththeCensus2000InternetForm.High V"! levelsofcustomersatisfactionindicateastrongpotentialforlargescaleimplementationin2010. <#" Keyrecommendationsinclude: %X $ " i "   iw?2#K3  0    ImplementacontentredesignoftheInternetcensushelpinstrument. iw??݌&$"&(#(# Ќ  " i "   ir@2#K3  0    Conductresearchonknowledgeandperceptionsofthedecennialcensusaswellasthe (#( needsofpotentialusersoftheCensus2000website. ir@@݌)$)(#(# Ќ  " i "   iA2#K3  0    Lookbeyondrestrictingonlineassistancetoquestionnairehelp. iAA݌R+&+(#(# Ќ   8,',  EvaluationA.3,BeCountedCampaignforCensus2000 (Carter2002a)  ThefourgoalsfortheBeCountedCampaigninCensus2000weretocountpersonswhodidnot | receiveacensusquestionnaire,tocountpersonswhobelievedtheywerenotincludedonany b othercensusform,toencourageparticipationofpersonswhoaretraditionallyundercountedin H thecensus,andtoprovideameansforpersonswithnousualresidencetobecounted. . ~ Respondentsreturned804,939BeCountedFormstotheCensusBureau.TheCensusBureau  J expectedapproximately1millionBeCountedFormstobereturned.Ofthe605,905BeCounted  0 Formsthatwereincludedincensusprocessing,239,128BeCountedFormsaddedpersonstothe    censusnotincludedonotherCensusforms.Therewere236,482householdswherethe    householdcontainedsomepersonswhowereonlyenumeratedfromtheBeCountedForm   return.Ofthesehouseholds,116,019wereenumeratedonlybyBeCountedFormsandthe x  remaining120,463wereenumeratedbyBeCountedFormsaswellasothercensusforms. ^  Therewere560,880personsaddedtothecensusthroughBeCountedForms.Thisismorethan *z  doublethenumberofpersonsaddedfromthe WereYouCounted?programinthe1990 ` Census.Therewerehigherpercentagesofgroupstraditionallyundercountedthanwereobserved F inthecensusoverall.Thesegroupsincluderenters,children,andminorities.Approximately , 40.7percentofallBeCountedFormsthatwerepickedupbyrespondentsfromdistributionsites  werenonEnglishforms,mostofwhichwereSpanish.Therewerealsoapproximately15,410  BeCountedFormsthatwerereturnedtotheCensusBureauthatweredeterminedtobefrom  personswithnousualresidence.ThismeansthatBeCountedFormsincreasedcoveragein t groupsthathavebeentraditionallyhardtocount. Z OveralltheBeCountedCampaignwasasuccess.Itadded560,880personstothecensus. &v Whilethisnumberissmall,thesearepeoplethatwouldhavebeenmissedwithoutthisprogram.  \ TheCensusBureaushouldconsiderthefollowingpointsifimplementinganoperationlikethe ( BeCountedCampaignin2010:  " i "   i-M2#K3  0    ConductfurtheranalysistoinvestigatethenumberofBeCountedFormsmatchedand   geocodedbytheautomatedsystemandbyclericalstaff. i-MZM݌p! (#(# Ќ  " i"   iN2#K3  0    DesigntheFieldVerificationtopermittheenumeratortorecordtheCensusidentification <#" number(ID)oftheBeCountedFormduplicates. iNN݌"$r#(#(# Ќ  " i"   iO2#K3  0    RecordthelanguageofthereturnedBeCountedFormsandwhethertranslationor %>!% transcriptionwasneeded. iOP݌&$"&(#(# Ќ   ' #'  EvaluationA.4,UseofNonEnglishQuestionnairesandGuidesintheCensus2000  LanguageProgram (SmithandJones2003 )   ForCensus2000,householdsinmailbackareasweremailedanadvanceletter.Theadvance b letterprovidedhouseholdsanopportunitytorequestanalternativelanguagequestionnaireinone H offivelanguages.ShortandlongformquestionnaireswereavailableuponrequestinSpanish, . ~ Chinese,Tagalog,Vietnamese,andKorean.Respondentswereaskedtoreturntheiradvance  d letterindicatingwhichlanguagequestionnairetheypreferred.Thisapproachspearheadedthe  J efforttoencouragerespondentsinlinguisticallyisolatedhouseholdstocompleteacensus  0 questionnaire.TheCensus2000LanguageProgramalsomadeavailablelanguageguidesin    fortyninelanguages.Thelanguageguideswereuserfriendlyvisualaidesthatassisted    respondentsincompletingthequestionnairesforbothlongandshortforms.   ThisevaluationanalyzedinformationaboutnonEnglishformsrequestedbyhouseholdswho ^  weremailedadvanceletters.TheresultsarecomparedtothenumberofrequestsforLanguage D  AssistanceGuides. *z  Therewereover2.2millionrequestsfornonEnglishquestionnaires.Most(83.7percent)ofthe F householdsthatrequestedalternativelanguagequestionnairesrequestedtheminSpanish. , AlthoughtherewererequestsfornonEnglishquestionnairesfromhouseholdsinallstates,most  (about70.6percent)oftherequestswerefromhouseholdsinfourstates:California,Texas,New  York,andFlorida.! !About39.4percentofthehouseholdsthatrequestedalternativelanguage  questionnaireswereincensustractsdesignatedashardtoenumerate.Lessthanhalf(45.1 t percent)ofthehouseholdsrequestinganalternatelanguageformreturnedtheseformsbymail. Z Thenumberofrespondentsrequestinglanguageassistanceguideswassubstantiallylowerthan &v thenumberofhouseholdsrequestinganonEnglishlanguagequestionnaire.Atleast93,672  \ respondentsrequestedlanguageassistanceguidesfromQuestionnaireAssistanceCentersand B 77,191respondentsrequestedlanguageassistanceguidesthroughtheTQAprogram.About34.3 ( percentofthelanguageassistanceguidesrequestedatQuestionnaireAssistanceCentersand18.9  percentofthelanguageassistanceguidesrequestedthroughtheTQAprogramwereinlanguages  otherthanSpanish,Chinese,Korean,Tagalog,orVietnamese.   Basedontheresultsofthisevaluation,therecommendationis: V"! " i"   iD]2#K3  0    Continuetotakeinitiativestohelprespondentsovercomelanguagebarriersin "$r# completingcensusformsbyprovidinganopportunityforhouseholdstomakerequests %X $ fornonEnglishquestionnaires,providingguidesandquestionnairesatassistancecenters, %>!% andprovidingalternatelanguagetelephoneassistance. iD]q]݌&$"&(#(# Ќ   (#(  EvaluationA.5.a,AnalysisoftheLinguisticallyIsolatedPopulationinCensus2000 (Lestina  2003)  Thefourobjectivesofthisevaluationweretocreateauniverseofhouseholdsidentifiedas b linguisticallyisolated,todeterminehowtheywereenumeratedinCensus2000,toexaminethe H educationattainmentofthehouseholder,andtoexaminegeographicclusteringatthetractand . ~ countylevels.  d Ahouseholdisclassifiedaslinguisticallyisolatedifallhouseholdmembersage14yearsor  0 olderspeakalanguageotherthanEnglishandhavelimitedEnglishproficiency.Thereare3,141    countiesinthenationwithatleastonelinguisticallyisolatedhousehold.Eachofthesehasupto    35.0percentofitshouseholdsthatarelinguisticallyisolated.Ofthe3,141counties,91.53   percenthavefewerthan5percentoftheirhouseholdsthatarelinguisticallyisolated.Thereare x  eightcountiesinTexaswithatleast25percentoftheirhouseholdsthatarelinguistically ^  isolated. D  Ofthe64,960tractsinthenationwithatleastonelinguisticallyisolatedhousehold,77.5percent ` havelessthan5percentoftheirhouseholdsthatarelinguisticallyisolated.Thereare11tractsin F thenationwhereatleast75percentoftheirhouseholdsarelinguisticallyisolated.Theyareas , follow:oneinMaricopaCounty,Arizona,oneinPinalCounty,Arizona,threeinLosAngeles  County,California,oneinSanFranciscoCounty,California,oneinLafourcheParish,Louisiana,  twoinBronxCounty,NewYork,oneinDutchessCounty,NewYork,andoneinCharleston  County,SouthCarolina. t Atthetractandcountylevels,thelinguisticisolationvariablemayhelpwithidentifyingareas @ forspecialenumerationprocedures,includinglanguageprograms,forthe2010Census.Further &v analysisshouldbedonebyspecificlanguagesthatarespokenathometoidentifytheleveland  \ whethertheyareclustered. B  EvaluationA.6.a,TheU.S.PostalServiceUndeliverableRatesforCensus2000Mailout  Questionnaires (Kohn2003)  InCensus2000,thequestionnairemailout/mailbackenumerationmethodwastheprimarymeans p!  ofcensustaking.TheU.S.PostalService(USPS)wastheprimaryvehiclefordeliveringcensus V"! questionnaires.BasedontheDecennialMasterAddressFile(DMAF),theCensusBureau <#" mailedquestionnairesonMarch1315,2000toabout96millionhousingunitsinareas "$r# designatedasbeingmailout/mailback.Questionnairesthatwereundeliverablewerecalled %X $ UndeliverableasAddressed(UAA).Sincethisstudyofquestionnairesconcernsundeliverability %>!% bytheUSPS,onlymailout/mailbackhousingunitsareincluded. &$"& TheCensus2000LocalCensusOffice(LCO)redeliveryoperationforUAAquestionnairestook (#( placeinpreselectedZIPcodesandwasconductedbyspeciallytrainedenumerators.TheUAA )$) questionnaireswereroutedbackthroughtheUSPSandreturnedforcheckinattheLCOuntil l*%* March18,2000.ByredeliveringquestionnairesidentifiedasUAAinareaswheretheywere R+&+ clustered,theCensusBureausoughttoefficientlyboostresponsebygettingquestionnairesback 8,', intothehandsofthehouseholdsearlyinthemailresponseperiodandtoaddressgeographic  clusteringofUAAquestionnaires.  AddressesremainingUAAafterthecensusredeliverywereincludedinthenonresponse b followup(NRFU)workload.DuringNRFU,manyhousingunitsforUAAquestionnaireswere H enumeratedasoccupiedhouseholds.Housingunitswhichweredeliveredaquestionnaireeither . ~ bytheUSPSorthecensusredeliverycouldhavereturnedtheirquestionnairesbymail.Those  d whichdidnotrespondbymailwerealsoincludedinNRFUandmanyofthesewerealso  J enumeratedasoccupiedhousingunits.  0 Themajorobjectivesofthisevaluationweretoexaminethedecreaseinundeliverablehousing    unitsasaresultoftheredeliveryoperationandtostudyrelationshipsbetweenUAAstatusand   demographicdata. x  Nationwide,theCensusBureaudeliveredtonearly600,000occupiedhousingunitsinthe D  redeliveryoperation.Ageofthehouseholder,tenure,andthesizeofthehouseholdarethebest *z  predictorsofUSPSdelivery.Minorityandnonminoritycountswereobtainedwiththeaidofa ` raceedit/allocationvariablewhichallocatedeachpersontooneandonlyoneofsixmajorrace F categories.MinorityhouseholdsweremorelikelytobeinLCOsselectedfortheredelivery , operationthannonminorityhouseholds.ForUSPSUAAunitsforwhichredeliverywas  attempted,nonminorityhouseholdsweremorelikelytohaveasuccessfulredeliverythan  minorityhouseholds.   EvaluationA.6.b,StudyoftheU.S.PostalServiceReasonsforUndeliverabilityofCensus Z 2000MailoutQuestionnaires (Chesnut2003b) @ DuringthemailoutofCensus2000questionnaires,theUSPSdesignatedeachquestionnaireas  \ UAAifitcouldnotbedeliveredsuccessfullytothelabeledaddress.Themailingpieceswere B annotatedwithareasonforundeliverabilityandsentbacktotheCensusBureau.Fromthe ( undeliverablequestionnairesreceived,astratifiedsystematicsamplewasdrawnforthepurpose  ofconductingastudyofthereasonsforundeliverability.Thisevaluationexaminedthe  distributionofreasonsforundeliverability.Inaddition,inferencesaredrawnfromthesampleto   theuniverseofCensus2000undeliverablequestionnairesatanationallevel.Thisevaluation p!  alsoinvestigatedwhetherthereasonforundeliverabilityisavalidindicatorofthefinalcensus V"! statusofahousingunit. <#" Fromtheresultsofthesamplingprocedure,atotalof9.7millionUAAquestionnaireswere %X $ receivedattheNationalProcessingCenterinJeffersonville,Indiana.Themostcommonreason %>!% questionnaireswerenotdeliverablewasduetothefactthatthehousingunitwasidentifiedas &$"& vacantbytheUSPS.Almosthalfoftheundeliverablequestionnairesreceivedwerestampedor ' #' annotatedwitha vacantreasonforundeliverability.TheUSPSpolicyisthatmailisnot (#( deliveredtovacantunits.Addressesidentifiedas nosuchaddressandthoseidentifiedasnot )$) havingamailreceptaclecomposedthenextlargestportionsofundeliverablequestionnaires. l*%* Theremainingundeliverabilitycategories(duplicate,underconstruction,demolished, R+&+ nonresidential,nosuchapartment,postoffice(P.O.)box,notdeliverableandunableto_forward, 8,', outsidedeliverylimits,refused,blankother,andillegible)eachcontributed6percentorlessto  theuniverseofundeliverablequestionnaires.  Thirtyonepercentofthetotalundeliverablequestionnairesreceivedafinalcensusstatusof b vacantand47percentreceivedafinalcensusstatusofdelete.Approximately22percentwere H givenafinalcensusstatusofoccupied.Forthequestionnairesmarked vacant,approximately . ~ 50percentactuallyreceivedafinalcensusstatusofvacant.Also,22percentofthe  d questionnairesmarkedvacantreceivedafinalstatusofoccupied.Forthequestionnairesmarked  J  postofficeboxor nomailreceptacle,anestimated48percentwereoccupied.This  0 highlightsthetroublesomeaspectofusing_mailout_/mailbacktoenumeraterespondentswhodo    notreceivemailattheirplaceofresidence.Inamuchmorefavorableresult,anestimated85    percentofquestionnairesmarked demolished, newconstruction,or nonresidential   receivedafinalstatusofdeleteandthat77percentofquestionnairesmarked nosuchaddress x  or nosuchapartmentreceivedafinalstatusofdelete. ^  Basedontheevaluationresults,recommendationsinclude: *z  " i"   i2#K3  0    UseUSPSproducts/servicessuchastheAddressElementCorrectionservicepriorto F Censusmailouttoprovidecorrectionstoaddressesortoidentifypotentially , undeliverableaddressesinthe_mailout_/mailbackaddresslist. i݌(#(# Ќ  " i"   il2#K3  0    CapturetheUSPSsreasonsforundeliverabilityandusetheseindeterminingfinal  censusstatus. il݌t(#(# Ќ  EvaluationA.7.a,Census2000MailResponseRates (StackhouseandBrady2003a) @ TheresponserateisameasurethatrepresentsthepercentageofaddresseseligibleforNRFUthat  \ returnedquestionnairespriortothedesignationoftheNRFUuniverse.Duetotheexpected B higherlevelofdataqualityandthelowercostassociatedwithselfenumeratedresponsesrelative ( toenumeratorcollectedresponses,itisimportantforresponseratestobeashighaspossible.  Themailresponserateisdefinedasthenumberofmailreturnsreceivedpriortothecutdatefor   theNRFUuniversedividedbythetotalnumberofhousingunitsinmailbackareasthatwere p!  eligibleforNRFU.Thefinalresponserateissimilarbutincludesallmailreturnsthroughthe V"! endoftheyear.Mailreturnsincludedintheresponseratesarepaperquestionnaires,interviews <#" duringtheTQAprogram,Internetdatacaptures,BeCountedForms,andCEFUreturns. "$r# ThemailresponserateasofApril18,2000was64.3percent.Thefinalresponseratewas67.4 %>!% percentasofDecember31,2000.Reflectingthehigherresponseburdenofthelongform &$"& questionnaire,theshortformmailresponserateof66.4percentwas12.5percentagepoints ' #' higherthanthelongformmailresponserateof53.9percent.Manyresidentswithlongforms (#( heldontothemandreturnedthemafterApril18.Afterthatdatealargerproportionoflong )$) formswerereturned_than_Ԁshortforms.Thefinalresponseratewas69.1percentforshortforms l*%* and59.4percentforlongforms. R+&+  8,', _Mailout/Mailbackareashadamailresponserateof65.4percent,whichishigherthaneitherthe  Update/Leave(U/L)areasmailresponserateof59.3percentortheUrbanU/Lareasmail  responserateof50.5percent.FinalresponseratesbyTypeofEnumerationArea(TEA)were | 68.5percentforMailout/Mailback,62.6percentforU/L,and54.8percentforUrbanU/L. b MostquestionnaireswerereturnedintheperiodbetweenMarch15,whenquestionnairesin . ~ Mailout/Mailbackareasweremailed,andMarch28.Therewereslightsurgesinthenumberof  d mailreturnscorrespondingtothedeliveryofreminderpostcardsbeginningonMarch20andon  J CensusDay(April1).Thesetwosurgesinresponseweremorepronouncedforlongformsthan  0 shortforms.    Anadditional1,052,712returnswerereceivedbetweenApril18andApril25,representing28.4   percentofthemailreturnscheckedinafterApril18.Thesereturnsrepresentapotential x  decreaseintheNRFUworkloadof2.5percent.Therefore,workneedstobedonetodetermine ^  whatistheoptimaldatefordeterminingtheNRFUuniverse,byconsideringthecostbenefits D  versustheoperationalchallengestootheroperations. *z   EvaluationA.7.b,Census2000MailReturnRates (StackhouseandBrady2003b) F ThemailreturnrateisameasureofrespondentcooperationinCensus2000.Itisdefinedasthe  numberofmailreturnsreceivedpriortothecutdatefortheNRFUuniversedividedbythetotal  numberofoccupiedhousingunitsinmailbackareasthatwereontheDMAFpriortoNRFU.  Thefinalreturnrateissimilarbutincludesallmailreturnsthroughtheendoftheyear.Mail t returnsincludedinthereturnratesincludeactualpaperquestionnaires,interviewsduringthe Z TQAprogram,Internetdatacaptures,BeCountedForms,andCEFUreturns. @ Themailreturnrateisdifferentfromthemailresponserate.Thedenominatorofthemail  \ responserateincludesallhousingunitsinmailbackTEAsthatwereeligibleforNRFUandhad B addressesthatwereconsideredadequatetoattemptdeliverybyeithertheUSPSorcensusfield ( staff.Theresponseratedenominatorislargerthanthereturnratedenominator,largelybecause  theresponseratedenominatorincludesvacanthousingunits,UAAaddresses,someaddresses  deletedinU/LandUrbanU/Ldelivery,NRFU,orCoverageImprovementFollowup(CIFU).   ThemailreturnrateasofApril18,2000was74.1percent.Thisraterepresents75,163,020mail V"! returnsthatwerereceivedbyApril18,2000outofareturnratedenominatorof101,398,131 <#" households.Another4,367,080questionnaireswerereturnedafterApril18,resultinginafinal "$r# returnrateasofDecember31,2000of78.4percent.Reflectingthehigherresponseburdenof %X $ thelongformquestionnaire,theshortformmailreturnrateof76.4percentwas13.4percentage %>!% pointshigherthanthelongformmailreturnrateof63.0percent.Manyresidentswithlong &$"& formsheldontothemandreturnedthemafterApril18.Afterthatdatealargerproportionof ' #' longformswerereturnedthenshortforms.Thefinalreturnratewas80.1percentforshort (#( formsand70.5percentforlongforms. )$) Mailout/Mailbackareashadamailreturnrateof75.1percent,whichishigherthaneitherthe R+&+ mailreturnrateforU/Lareas(69.6percent)orthemailreturnrateforUrbanU/Lareas(63.7 8,', percent).FinalreturnratesbyTEAwere78.6percentforMailout/Mailback,77.9percentfor  U/L,and70.8percentforUrbanU/L.  Differentialreturnrateswereobservedfordifferentdemographicgroups.Thelikelihoodof b respondingtothecensusincreasedwithhouseholdersage.Whiteshadahighermailreturnrate H (77.5percent)thanthetotalmailreturnrate,whileallotherracegroupshadlowerreturnrates . ~ thanthetotalmailreturnrate.NonHispanichouseholdershadamailreturnrateof75.0percent,  d 10.5percentagepointshigherthantheHispanicmailreturnrateof64.5percent.Households  J consistingoftwopersonshadthelargestproportionofresidentswhorespondedtothecensus.  0 Largerhouseholdsoffivepersonsormorehadincreasinglylowermailreturnratesashousehold    sizeincreased.     EvaluationA.8,PuertoRicoFocusGroupsonWhyHouseholdsDidNotMailBackthe x  Census2000Questionnaire (Berkowitz2001b) ^  Census2000wasthefirsttimethatresidentsofPuertoRicowereaskedtocompleteandreturn *z  theirquestionnairesbymail.FiftythreepercentofthehouseholdsinPuertoRicoreturnedtheir ` questionnairesbymail,alowresponseratecomparedwiththenationalrateof65percent.This F evaluationexploredthereasonsresidentsofPuertoRicodidn'tmailbacktheirquestionnaires. , Contractorsconductedfocusgroupsinninesites.Siteswereselectedforgeographicand  socioeconomicdiversityfromamongmunicipalities(municipios)withlowerthanaverage  (under50percent)mailbackresponserates.Althoughinterestingandsuggestive,theseresults  arebasedonasmall,purposivesampleandcannotbegeneralizedtothepopulationofPuerto t Ricoasawhole.However,theydoprovideausefuljumpingoffpointforfurtherthinkingand Z research. @ Participants'reasonsfornotreturningtheirquestionnairesbymailfallintofourclusters:  \  * #$%&'((xx #$%&'(0  Motivationalandprocessrelatedreasons.Severalparticipantswereunclearaboutor ( misunderstoodthecensus'purpose.Considerableconfusionexistedovertheprocessby  whichthequestionnairesweredistributed,therulesforreturningthem,andespeciallythe  roleofenumerators. (#(#  #$%&'((#Kxx* #$%&'(0  Practicalandlogisticalreasons.Lackoftimefiguredasareason,asdiddifficultiesof V"! mailingandgettingtothepostoffice.<#"(#(# 0  Culturalandpoliticalattitudinalreasons.Theseincludedfearsthattheinformation %X $ wouldn'tbekeptconfidential,aswellasthebeliefthatanyfundsthatwouldbeallocated %>!% wouldonlyendupenrichingdishonestpoliticians.Participantsalsoexpressedastrong &$"& preferenceforamorepersonalapproachtocollectingtheinformationasbeingmore ' #' appropriatetotheculture.(#((#(# 0  Reasonsrelatedtoquestionnairecontentanddesign.Someparticipantswereaffronted l*%* bytheraceandethnicityquestions,whichwereseenasdivisiveaswellasinappropriate R+&+ totherealitiesofPuertoRico.8,',(#(# ЇAlmosteveryonehadheardsomethingaboutCensus2000fromtelevisionandradioads,  newspapers,schools,orinformalsourcessuchasrelativesandneighbors.Butinmanycases,  advertisingeffortsfellflat.Nearlyallofthe41nonrespondinghouseholdheadsurgedareturn | tothesystemofcollectingdatadoortodooraspracticedin1990. b Keyrecommendationsforimprovementsandforfurtherresearchandexplorationinclude: . ~ " i"   i42#K3  0    Providemorecomprehensiveinformation,indifferentforms,onthepurposeandusesof  J thecensus. i4a݌ 0(#(# Ќ  " i"   ia2#K3  0    ConductfurtherresearchinPuertoRicoonviewsandperceptionsoftheCensus2000    questionsonraceandethnicity. ia݌ (#(# Ќ  " i"   i2#K3  0    Usethefindingsandrecommendationsfromthisstudy,aswellasothersupporting ^  testingandresearch,todevelopadvertisingappealsforcensusparticipationmoreattuned D  toPuertoRico. iγ݌*z (#(# Ќ  " i"   i<2#K3  0    Considerusingtheresultsofthisstudytocreateacloseendedsurveytobeadministered F toaprobabilitysampleofresidentsofPuertoRico. i<i݌,(#(# Ќ  CategoryBContentandDataQuality   SummariesoftheeightevaluationsintheContentandDataQualitycategoryfollow. Z  EvaluationB.1.a,AnalysisofImputationRatesforthe100PercentPersonandHousing &v UnitDataItemsfromCensus2000 (Zajac2003)  \ Thisevaluationprovidesinformationondataquality,specificallydatacompleteness,forthe100 ( percentpersonandhousingunititems.Forthisevaluation,datacompletenessismeasuredby  imputation.Itshouldbenotedthatthedefinitionofimputationcanbeinterpretedinvarious  wayswhichcouldleadtodifferentmethodsofcomputingimputationrates.Therefore,when   comparingimputationratesacrossreports,itisimperativetounderstandthewaytheratesare p!  computedtoensuretheyarecomparable. V"! Imputationisdividedintothreecategories.Theyaredefinedasfollows: "$r#  * #$%&'((xx #$%&'(" i"   iR23  0    Anassignmentisperformedwhenaresponseforadataitemiseithermissingornot %>!% consistentwithotherresponsesandanitemvaluecanbedeterminedbasedon &$"& informationprovidedforthatsameperson. iR݌' #'(#(# Ќ  " i"   i23  0    Allocationsareperformedwhenaresponseforadataitemiseithermissingorisnot )$) consistentwithotherresponsesandanitemvaluecannotbedeterminedbasedon l*%* informationprovidedforthatsameperson.Anallocationusesaresponsefromanother R+&+ personwithinthehouseholdorfromapersoninanearbyhousehold.Whenevery100 8,', percentcharacteristicforapersonrequiresallocation,thepersonisconsideredtotally  allocatedwhenatleastonepersonwithinthehouseholdhasdatathatdonotrequire  allocation.However,wheneveryitemforeverypersoninthehouseholdrequires | allocation,itiscoveredbysubstitution. i/݌b(#(# Ќ  " i"   i23  0    Asubstitutionoccurswhenallthe100percentcharacteristicsforeverypersoninthe . ~ householdareeithermissingorarenotconsistentwithotherresponses.Anearby  d housingunitwithcomplete100percentdataisselectedtorepresentthemissingor  J inconsistentdataitems.Thisnearbyhousingunitisselectedusingthenearestneighbor  0 hotdeck.Thisisalsocalledawholehouseholdsubstitution. i݌  (#(# Ќ  Inadditiontothesethreetypesofimputationrates,adatacompletenessstatisticwasproducedto   determinethenumberof100percentpopulationitemswithineachpersonrecordthatwerenot x  imputed. ^  Almost1.5millionhouseholdsweresubstituted,representing1.39percentoftheoccupied *z  housingunits.Withinthesesubstitutedhouseholds,therewereover3.4millionsubstituted ` persons,accountingfor1.26percentofthepersonsinhousingunits. F Totalitemimputationratesforthe100percentpersondataitemsrangefrom1.98percentforthe  sexitemto5.08percentfortheageitem.Thetenureitemimputationratewas5.48percent.In  general,shortformdataweremorecompletethanlongformdata,selfresponsedataweremore  completethanenumeratorreturndata,Englishformshadmorecompletedatathanforms t designedforotherlanguages.Forallitems,dataforownersweremorecompletethanfor Z renters. @ <(4 4 <DL!X<Overall,thedatacompletenessstatisticshowsthatabout97percentofnonsubstitutedperson  \ recordshaveatleastfourofthefive100percentpopulationitemswithnonimputeddata.B+ 4 <DL!44X!B B  EvaluationB.1.b,AnalysisofItemNonresponseRatesforthe100PercentHousingand  PopulationItemsfromCensus2000 (Norris2003)  ThisevaluationdeterminedtheextentofitemnonresponsefortheCensus2000hundredpercent p!  items.Inconsistentresponsesarenotconsiderednonresponse.Rateswerereportedforeachof V"! the100percenthouseholdpopulationitemsandtenureclassifiedbyformtype(longversus <#" short)andresponsemode(selfversusenumerator).Inaddition,resultsoftheInternetreturnsare "$r# reported.Someofthebreakdownswithinreturncharacteristicsaresubjecttointerpretation. %X $ XzXXX#XzXXXzS#Thedefinitionofitemnonresponseissometimesinterpretedinvariouswaysdependingonthe &$"& scopeofaparticularanalysis.Thiscouldleadtodifferentmethodsofcomputingitem ' #' nonresponseratesandmaypotentiallyleadtoconflictingratesbetweenreports.Therefore, (#( whencomparingitemnonresponseratesacrossreports,itisimperativetounderstandthewaythe )$) ratesarecomputedtomakesurethattheyarecomparable. l*%* Itemnonresponseforthe100percentitemsrangedfrom1.13percentforthesexitemto_4.14 8,', percentforthetenureitem.Generallyitemnonresponsewashigherforenumeratorreturnsthan  forselfresponsesandhigherforlongformsthanforshortforms.Tenurehadarelativelyhigher  overallnonresponseratecomparedtootheritems. | Recommendationsinclude: H " i"   i23  0    Reviewtheproceduresanddebriefingsoffieldstafftoseeiftheycanprovideuseful  d informationaboutproblemsthatcouldhaveledtoitemnonresponse. i݌ J(#(# Ќ  " i"   i23  0    Continuetotestquestionwordingandplacement. i݌  (#(# Ќ  " i"   i23  0    Investigatewaystoreduceitemnonresponse. i݌ (#(# Ќ  " i"   i23  0    ReviewtheresultsofEvaluationB.1.ainconjunctionwiththeresultsofthisevaluation ^  toobtainamorecomprehensiveviewofdataquality. i݌D (#(# Ќ  " i"   i23  0    Investigatetheuseofcontentfollowupforrelativelyhighitemnonresponseitems. i?݌`(#(# Ќ  " i"   i23  0    Lookattheageitemincombinationwiththedateofbirthitemduringfutureanalysis. iI݌,(#(# Ќ  EvaluationB.3,CensusQualitySurveytoEvaluateResponsestotheCensus2000Question  onRace:AnIntroductiontotheData (Bentleyetal2003)  Dataonracefrommostfederalsurveyscurrentlyreflectacollectionmethodologythatasks Z respondentstomarkonlyonecategory.Census2000wasthefirstdecennialcensustoask @ respondentsto markoneormoreraces.Somedatausersmaywanttocomparetherace &v distributionfromCensus2000withthoseofotherdatasourceswhererespondentswereaskedto  \ markonlyoneraceforeachpersoninahousehold.TheOfficeofManagementandBudget B referstothiscomparisonas bridging. ( TheCensusQualitySurveyenablesuserstomakecomparisonsbetweenracedataobtainedusing   markoneraceand markoneormoreracesmethodsbycollectingracedatausingboth   methodsfromthesamepeople.TheCensusQualitySurveywasdesignedwiththeprimary p!  objectiveofproducingadatafilethatcouldbeusedtobridgebetween singleand oneormore V"! racesdistributions.TheCensusQualitySurveyhadanationallyrepresentativedesignwithtwo <#" datacollectionpoints.Respondentswereaskedatonepointto markoneraceandatanother "$r# pointto markoneormoreraces.Thesamplewassplitintotwopanels.PanelAreceivedthe %X $  markoneormoreracesinstructionattheinitialcontactandPanelBreceivedthe markone %>!% raceinstructionfirst.Duringtherecontact,eachpanelreceivedthealternateinstruction.Data &$"& fromthesetwocontactscanbeusedtoproduce bridgingparameterstocomparerace ' #' distributionscollectedundersingleraceandoneormoreracemethodologies. (#( Initially,about27,500housingunitaddressesweredesignatedtobeinsampleforeachpanel. l*%* Oftheeligibleaddresses,97percentcompletedaninterviewintheinitialcontact.Inthe R+&+ recontact,samplehousingunitswerecontactedonlyifaninitialquestionnairewascompleted. 8,', Oftheeligiblerecontactaddresses,87percentcompletedaninterviewinPanelAand94percent  completedaninterviewinPanelB.Theresultsfromthequestiononracesuggestthateach  panelappearstoberepresentativeofracedistributionsinCensus2000. | FortypercentofthenonHispanicrespondentsinPanelAwhoreportedTwoorMoreRacesin H Census2000alsoreportedTwoorMoreRacesintheinitialcontact.Similarly,41percentof . ~ thoseinPanelBwhoreportedTwoorMoreRacesinthecensusalsoreportedTwoorMore  d Racesintherecontact.Theeffectivesamplesizeforcomputingbridgingparametersisreduced  J becauseofthegenerallylowlevelofconsistencyinthereportingofTwoorMoreRaces.In  0 contrast,97to98percentofthosewhoreportedasingleraceofWhite,Black,orAsianin    Census2000reportedthesameraceintheCensusQualitySurvey.    The markoneormoreracesdatacollectioncontactwascrosstabulatedwiththe markone x  racecontacttoassesshowindividualsrespondwhenaskedtochooseasingleraceforpeople ^  forwhommultipleraceshavebeenreported.Evenwiththe markoneraceinstruction,a D  significantportionofrespondentsreportedTwoorMoreRaces.Thisportionwasgreatly *z  reducedwhenthefollowupraceprobewasusedinthePanelArecontact.Usersofthedatafile ` willneedtodeterminehowbesttotreatthesereluctantcaseswhencomputingbridging F parameters. ,  EvaluationB.5,ContentReinterviewSurvey:AccuracyofDataforSelectedPopulation  andHousingCharacteristicsasMeasuredbyReinterview (SingerandEnnis2003)  TheContentReinterviewSurveywasdesignedtoevaluatetheconsistencyofresponsestothe Z Census2000questionnaire.Previouscontentreinterviewsurveysattemptedtoevaluateboth @ responsevariance(thevariationinresponsesoverrepeatedquestioning)andbias.In2000,only &v responsevariancewasstudied.Toreducecostandtheburdentorespondents,the2000Content  \ ReinterviewSurveyaskedpopulationquestionsaboutonlyonesamplepersonperhousehold, B whowasrandomlychosenfromarosterforeachunitthatwascollectedatthebeginningofthe ( ContentReinterviewSurvey.  PriortoCensus2000enumeration,30,000householdsthatwereinitiallyselectedtoreceivethe   censuslongformquestionnairewererandomlyselectedaspotentialparticipantsintheContent p!  ReinterviewSurvey.Afterahouseholdreturnedthecensusquestionnaire,itbecameeligibleto V"! participateinthereinterviewsurvey.Experiencedcensusfieldrepresentativescalledthe <#" selectedhouseholdstoreaskmostofthecensuslongformquestions.Personalvisitinterviews "$r# wereallowedifthehouseholdscouldnotbereachedbytelephone. %X $ FortheContentReinterviewSurvey,datawereanalyzedfromabout20,000ofthepreselected &$"& households.Aroundthreequartersofthecasesanalyzedhadcompletedthemailbackformsfor ' #' Census2000.AboutthreefifthsofallpreselectedreinterviewhouseholdscompletedCensus (#( 2000mailbackforms. )$)   R+&+ SincetheContentReinterviewSurveywasconductedbyenumeratorswhousedeithertelephone  interviewsorpersonalvisits,collectionmodeforthereinterviewsurveywasdifferentfromthat  ofthecensusinthemajorityofanalyzedcases. | Basedondatacollectedinthecensusandthereinterviewsurvey,analystscomputedtheindexof H inconsistency(ameasuretodetectresponsevariance)andusedittoevaluatetheconsistencyof . ~ eachitematthenationallevel.Ahighindexofinconsistency(50ormore)foraquestion  d indicatedthatthequestionwasproblematicbecausethedataelicitedbythequestionwasnot  J consistent.Alowindex(below20)indicatedthatthedataelicitedbythequestionwasprobably  0 consistent.Amoderateindex(20upto50)indicatedthatthequestionwassomewhat    problematic.    Ofthe58populationcharacteristicsevaluatedbytheContentReinterviewSurvey,16showed x  lowinconsistency,26showedmoderateinconsistency,and16showedhighinconsistency.The ^  itemsthatshowedlowinconsistencyincludedquestionsaboutsex,age,Hispanicorigin,marital D  status,schoolattendance,languagespokenathome,placeofbirth,citizenship,yearofentryto *z  theU.S.,andveteranstatusandperiodofmilitaryservice.Theitemsthatshowedhigh ` inconsistencyincludedquestionsaboutlanguageusage,disability,grandparentsascaregivers, F workexperiencein1999,andincome.Forthefirsttimeever,Census2000allowedthe , respondenttochooseoneormoreracesinresponsetotheracequestion.Theeditedracedata  displayedmoderateinconsistency.  Ofthe36housingcharacteristicitemsmeasured,fiveshowedlowinconsistency,15showed t moderateinconsistency,and16showedhighinconsistency.Theitemswithlowinconsistency Z includedquestionsaboutthenumberofpeopleinthehousehold,whethertheunitwasownedor @ rented,heatingfuel,whethertherewasamortgageontheproperty,andifrealestatetaxeswere &v includedinthemortgagepayment.Theitemswithhighinconsistencyincludedquestionsabout  \ utilitycostsforgasandforelectricity,secondmortgagesandhomeequityloans,loansonmobile B homes,thevalueofthepropertyandinsurancecostsfortheproperty,thenumberofroomsinthe ( house/apartment/mobilehome,whethertherewasabusinessatthesiteandthetotalvalueof  agriculturalsalesfortheproperty,andplumbingfacilities,kitchenfacilities,andtelephone  service.   Sufficientdataweregatheredtocompareindexesofinconsistencybycollectiontypefor87 V"! items.Atthe90percentconfidencelevel,51itemsshowedlessinconsistencyformailback <#" formsandtwoshowedlessinconsistencyforenumeratorforms.Thetwothatwereless "$r# inconsistentwhencollectedbyenumeratorswere DoyouspeakalanguageotherthanEnglish %X $ athome?and WhatistheannualcostforGas? %>!% Keyrecommendationsfollow: ' #' " i "   i23  0    Usecognitiveexpertstorecommendimprovementstoproblematicquestions. i݌)$)(#(# Ќ  " i "   i23  0    Useresultsfromcontenttestsindevelopingquestionnairesforthe2010Censusandthe R+&+ AmericanCommunitySurvey(ACS). i݌8,',(#(# Ќ  _" i "   i23  0    Providebetterinstructionsonthe2010CensusandtheACSfortheHispanicorigin  question. iC݌(#(# Ќ  " i "   i>23  0    Useseparate Yes/Noquestionsforeachresponsecategoryof markallthatapply b questions. i>k݌H(#(# Ќ  " i "   is23  0    Createadatabasethatlinkschangesinidentifiersforcensuscasestoenablelocating  d ContentReinterviewSurveycaseswhentherearechangestoMasterAddressFile(MAF)  J IDs. is݌ 0(#(# Ќ  " i"   i23  0    Planthecontentreinterviewsurveysofthe2010CensusandtheACSasearlyaspossible    andconductthemwithinthreeorfourweeksofcompletingtheoriginaldatacollection. i1݌ (#(# Ќ  " i"   i23  0    Totheextentpossible,usethesamedatacollectionmodes,datacapturemethods,data ^  processingprocedures,andenumeratorsforboththe2010Censusanditscontent D  reinterviewandforboththeACSanditscontentreinterview. i݌*z (#(# Ќ  " i"   iC23  0    Fortimesensitivequestions,refertothedateoftheoriginalsurveyinthecontent F reinterviewforboththeACSandthe2010Census. iCp݌,(#(# Ќ  EvaluationB.6,MasterTraceSample (HillandMachowski2003)  TheMasterTraceSampledatabaseprojectmergedCensus2000datafrommultiplesourcesto t provideinformationaboutcasesinthevariousphasesofdatacollectionandprocessing.The Z objectiveofthiseffortwastosupportfuturemethodologicalandoperationalanalysesand @ decisionsregardingthe2010Censusbycreatingacomplex,relationaldatabaseforresearch &v purposes.TheprototypedatabasemergedCensus2000addressframe,collection,enumeration,  \ capture,processing,response,andcoveragefiles.Thismergeyieldsasophisticateddatabase B whichallowsquantitativeinsightintotherelationshipofkeycensusprocesses.Inadditionto ( beinganinnovativeresearchtool,theCensus2000MasterTraceSampledatabaseisintendedto  serveasamodeluponwhichtheCensusBureauwillimproveinfuturecensuses.  TheMasterTraceSampledatabasecontainsasampleofCensus2000housingunitrecordsthat p!  allowCensusBureauresearcherstotraceresponseandoperationaldatathroughstagesofCensus V"! 2000processing.Thesestagesincludeaddresslistdevelopment,datacollection,datacapture, <#" anddataprocessing.Forthesampleofhousingunitrecords,thedatabasecontainsallreturns, "$r# whichinclude100percenthousingunitandpersondata. %X $ TheMasterTraceSampledatabasealsocontainsdatanottypicallyanalyzedincensus &$"& evaluations.Forexample,thenumberoftimesanenumeratorvisitedahousingunitduring ' #' nonreseponsefollowupiscontainedinthisdatabase.Inaddition,thedatabaselinksmicrolevel (#( datasuchasenumeratorproductiondatawithresponsedata,whicharenottraditionallylinkedin )$) censusevaluations.ThepurposeoftheMasterTraceSampledatabaseistofacilitateresearchon l*%* relationshipsamongCensus2000operationsbeyondthecurrentCensus2000Testing, R+&+ Experimentation,andEvaluationProgram. 8,', ЇThedatabasecontainsatotalofapproximately1.5millionMAFhousingunitIDsfroma  systematicIDsampleandablockclustersample.Theblockclustersamplecontainsallhousing  unitIDswithinselectedblockclusters. | Theprimaryrecommendationsare: H RecommendationsforExpandingtheCensus2000MasterTraceSampleDatabase:  d " i"   i23  0    ExpandtheMasterTraceSampletoincludedataonGroupQuarters. i݌ 0(#(# Ќ  " i"   i23  0    ExpandtheMasterTraceSampledatabasetoincludecoveragemeasurementdata    associatedwithpersons. i݌ (#(# Ќ  Recommendationforthe2010CensusMasterTraceSampleDatabase: ^  " i"   i23  0    Implementaformalevaluationtoassessboththeusefulnessofthedatabaseforresearch *z  andthebenefitstotheCensusBureauofresultingproducts. i<݌`(#(# Ќ  RecommendationforCreatingMasterTraceSampleDatabasesforotherCensusesandSurveys: , " i"   i23  0    ProvidedtheproposedformalevaluationoftheCensus2000MasterTraceSamplefinds  thedatabaseuseful,theCensusBureaumaywishtoconsiderbuildingsuchatrace  databasespecifictoeachofitsmajorsurveys,aswellastheEconomicCensuses. i ݌t(#(# Ќ  EvaluationB.7,AccuracyofDataforEmploymentStatusasMeasuredbytheCPS @ Census2000Match (PalumboandSiegel2004) &v ThisevaluationpresentstheresultsofanexactmatchstudythatusedtheCurrentPopulation B Survey(CPS)Census2000MatchtoevaluatethelaborforcedatainCensus2000bymaking ( estimatesoftheircontenterror(contenterrorreferstotheaccuracyofthedata,asopposedto  coverageerror,whichreferstohowcompletelypeopleandhousingunitsarecounted).The  evaluationcontainsadescriptionofthemethodsusedtocreatethematchfileandadescription   ofhowthefilewasusedtomeasurelevelsofcontenterror. p!  ForpeopleinCensus2000whowerealsointheCPSsampleinFebruarythroughMay2000,the <#" CPSCensus2000Matchbroughttogethereachpersonscensusreportwiththesamepersons "$r# CPSreport.Thislinkageprovidedtheopportunitytocomparetwoindependentobservationsof %X $ thesameevent(thepersonsrelationshiptotheworkforceataparticulartime)andtousethe %>!% outcomeoftheoneobservation(thepersonslaborforceclassificationintheCPS)tomeasure &$"& theaccuracyoftheoutcomeoftheother(thesamepersonslaborforceclassificationinCensus ' #' 2000).TheCPSwasusedbecauseitisconsideredtobethestandardofcomparisonforcensus (#( laborforcedata.Theanalysisinthisevaluationwasrestrictedtothenationallevel. )$) Crosstabulationofthetwoobservationspresentsestimatesofthepotentialquantitiesof R+&+ responseerrorinpublishedcensusfigures.Aresponseerrorissaidtooccurwhena_persons 8,', laborforceclassificationinCensus2000aseitheremployed,unemployed,ornotinlaborforce  differsfromthatsamepersonsclassificationintheCPS.Tomakethesequantitiesmeaningful,  tworelativemeasuresofresponseerrors(percentagedistributions)andtwosummarymeasures | ofresponseerrorswerederivedfromthem.Thederivedmeasuresarethefocusofthe b evaluation.ThepercentagedistributionsrevealthesuccessratesofCensus2000inclassifying H peopletotheircorrect(sameasCPS)laborforcecategoriesandawayfromincorrect(different . ~ fromCPS)categories.  d FindingsfromtheCPSCensus2000Matchinclude:  0  * #$%&'((xx* #$%&'(" i"   iI%23  0    Census2000andtheCPSarereasonablyconsistentinclassifyingpeopletotheemployed    andnotinlaborforcecategories,buttheyexhibitconsiderablevariabilityinclassifying   peopletotheunemployedcategory. iI%v%݌x (#(# Ќ  " i"   i&23  0    PreviousstudiesofcensusCPSemploymentclassifications,whichweredoneforthe D  1960and1970censusesbutnotthe1980and1990censuses,revealedpatternssimilarto *z  thosedescribedabove.However,forCensus2000,consistencyslippedsomewhatfrom ` the1970levels,inspiteofefforts,particularlyafterthe1990Census,tomakethecensus F employmentquestionsconformmorecloselywiththeCPSquestions. i&+'݌,(#(# Ќ  " i"   i)23  0    Aswastrueinthe1970and1960studies,theindexofinconsistencymeasuresforthe  unemployedcategorywereinthehighrange(above50),whichcallsintoquestion  whethertheunemployedconceptismeasurableinacensuscontext. i))݌t(#(# Ќ  " i"   ij+23  0    Theresultsfortheemployedandnotinlaborforcecategoriesindicatedthat,althoughthe @ censusisabletomeasuretheseconceptsreasonablywell,improvementsareneeded.The &v studysuggested,forexample,thatitmayhavebeenamistaketousetheCPSwordingfor  \ the worklastweekquestioninCensus2000. ij++݌B(#(# Ќ  " i"   i-23  0    Theunderestimateofemploymentandtheoverestimateofpeoplenotinthelaborforce  inCensus2000relativetotheCPSislikelyrelatedtothe_failure_Ԁofthecensus  classificationsystemtofiltermoreemployedpeopleoutofthenotinlaborforcecategory   andintotheemployedcategory. i--݌p! (#(# Ќ  " i"   i/23  0    ThedifferencebetweenthereferenceperiodsforthelaborforceestimatesofCensus <#" 2000andtheCPSisprobablynotamajorcontributortothegapsbetweentheestimates. i/0݌"$r#(#(# Ќ  " i"   iM123  0    AtendencyforpeopleclassifiedasemployedintheCPStobeclassifiedasnotemployed %>!% inCensus2000appearedtobeassociatedwithparticularagecategories,classofworker &$"& categories,andeducationalattainmentcategories.Thefindingsuggeststhatsomegroups ' #' ofworkersmayhavehaddifficultyinunderstandingorcorrectlyrespondingtothework (#( lastweekquestioninthecensus. iM1z1݌)$)(#(# Ќ   R+&+  EvaluationB.12,PuertoRicoCensus2000ResponsestotheRaceandEthnicityQuestions   (Christenson2003)  PriortoCensus2000,racedatawerelastcollectedinPuertoRicobyenumeratorsinthe1950 b CensusofPuertoRico.HispanicoriginhadneverbeforebeenaskedinPuertoRico.InCensus H 2000,thequestionsonraceandHispanicorigininPuertoRicowereidenticaltothequestions . ~ askedinthe50statesandtheDistrictofColumbia(DC).Boththenewnessofthedataandthe  d uncertaintiesassociatedwiththeracequestioncontributedtoahighlevelofinterestinthe  J responsesbyresidentsofPuertoRicotothesequestions.  0 Thegoalofthisstudywastoexaminethedataresultingfromtheresponsestotheraceand    HispanicoriginquestionsbytheresidentsofPuertoRicoandtocomparethemwiththose   resultingfromresponsestotheraceandHispanicoriginquestionsbyresidentsofthe50states x  andDC. ^  TheanalysisshowsthattheresidentsofPuertoRicoidentifiedthemselvesasoverwhelminglyof *z  Hispanicoriginandofasinglerace.Intermsofrace,thegreatmajorityidentifiedthemselvesas ` White,withasubstantialminorityreportingthemselvesasBlackorAfricanAmerican.When F comparedwiththoseofHispanicorigininthe50statesandDC,substantiallyfewerreported , themselvestobeofSomeOtherRace,andalowerpercentageidentifiedthemselvesasofTwoor  MoreRaces.Theanalysisalsoshowsthattwotypicalindicatorsofproblematicquestions,item  nonresponseanddifferencesbetweenrespondentandenumeratorcompletedquestionnaires,did  notindicatemajorproblemswitheitherquestion. t Fromthesefindingscomethefollowingrecommendationsforimprovementsandforfurther @ research. &v " i"   i<23  0    IncludePuertoRicointhecognitivetestingoreffortstofieldtestdifferentversionsand B formatsofquestionsandquestionnairesthatarecurrentlyunderwayforthe2010Census. i<<݌((#(# Ќ  " i"   i>23  0    Investigatefurthertheuseofthe SomeOtherRacecategorytoassurethattherangeof  responsesexcludesthosethatbetterfittheHispanicoriginquestion. i>/>݌ (#(# Ќ  EvaluationB.13,PuertoRicoFocusGroupsontheCensus2000RaceandEthnicity V"! Questions (Berkowitz2001a) <#" ThisevaluationexploredtheviewsandperceptionsofresidentsofPuertoRicoontheCensus %X $ 2000PuertoRicoshortformmailbackquestionnaireitemsonraceandHispanicorigin. %>!% Contractorsconductedfocusgroupsin12sitesacrosstheislandselectedforgeographicand &$"& socioeconomicdiversity,recruitingparticipantsofdiverseagesandeducationallevels,including ' #' somewhohadlivedintheU.S.foranextendedperiod.Althoughtheseresultsarebasedona (#( relativelysmall,purposivesampleandcannotbegeneralizedtothePuertoRicanpopulationasa )$) whole,theyprovideaninterestingjumpingoffpointforadditionalthinkingandfurtherresearch. l*%*  8,', _Therewasunanimousagreementamongfocusgroupparticipantsthatthequestiononraceis  inappropriatetothePuertoRicancontext.Participantscouldnotfindthemselvesreflectedinthe  availableanswercategories,whichtheyviewedasforeigntothePuertoRicoshistoryand | cultureofmixingorblendingacrossracialgroups.Somealsofeltthequestionwasinherently b racist,discriminatory,anddivisive,andsuspectedahiddenpoliticalagenda.Takingtheirlead H fromthenationalorigintermsinthesecondhalfofthequestion,someparticipantschoseto . ~ defineraceasnationality.However,theywerebotheredthat PuertoRicandidnotappearasa  d preprintedcategory.Participantswerenotsatisfiedwiththeoptionofcheckingoffmultiple  J racialcategoriesforanindividual,becausetheydidnotperceivethemselvesasbiracialor  0 multiracialbut,rather,asmixed.    Thepresenceofapreprinted Yes,PuertoRicananswercategoryrescuedtheHispanicorigin   questionfromthesamefateasthequestiononrace.Ingeneral,participantsinterpretationsof x  thisquestionwerehighlycontextdependent.Theydisagreedastowhethertheterms Spanish, ^   Hispanic,and Latinoallmeantthesamethingandintheirassessmentsofitsusesasa D  blanketcategory. Originwasvariouslyunderstoodasbirthplace,ancestry,nationality,and *z  selfidentification.Intheend,participantsdecidedthisquestionwasnotnearlyassimpleand ` straightforwardasithadfirstappeared. F Ontherelationshipofthetwoquestions,thelargestclusteroffocusgroupparticipantsviewed  themassoredundantthatitmadenosensetoaskbothandfavoredeliminatingtheracequestion  asthemoreoffensiveandlessinformativeofthetwo.  Keyrecommendationsinclude: Z " i "   iL23  0    IncluderesidentsofPuertoRico(inadditiontopersonsofPuertoRicanoriginlivingin &v theU.S.)inanysubsequentcognitivetestingoreffortstofieldtestdifferentversionsof  \ questionsandquestionnairesforfuturecensuses. iLLL݌B(#(# Ќ  " i "   iM23  0    ProvidemoreextensivepubliceducationinPuertoRicoonthelargermissionofthe  census,therationaleforaskingquestionsaboutraceandethnicity,andtheintendeduses  ofthedata. iMN݌ (#(# Ќ  " i "   iO23  0    Considerusingtheresultsofthisstudy,alongwiththeresultsofEvaluationA.8tocreate V"! asurveytobeadministeredtoaprobabilitysampleofresidentsofPuertoRico. iOO݌<#"(#(# Ќ  CategoryCDataProducts  %>!%  EvaluationC.1,TheEffectsoftheDisclosureLimitationProcedureonCensus2000 ' #' TabularDataProducts,Abridged (SteelandZayatz2003 )  (#( Thefullreportforthisevaluationisnotavailablebecauseitcontainsproprietaryinformation. l*%* MostinformationinthefullevaluationisCensusConfidential.Thefullevaluationcannotbe R+&+ removedfromCensusBureaufacilitiesandisavailabletoCensusBureaupersonnelonaneed_Ԅ 8,', Їtoknowbasis.Abridgedinformationisprimarilydescriptiveandqualitative.Quantitative  informationcanonlybefoundintheunabridgedevaluation.  DataswappingwasusedtoprotecttheconfidentialityoftheCensus2000tabulations.The b procedurewasperformedontheunderlyingmicrodataandalltabulationsfromthe100percent H (shortform)andfromthesample(longform)datawerecreatedfromtheswappedfiles.It . ~ affectedpairsofhouseholds(orpartneredhouseholds)whereoneorbothofthosehouseholds  d hadahighriskofdisclosure.Thesetofcensushouseholdsthatweredeemedashavinga  J disclosureriskwasselectedfromtheinternalcensusdatafiles.Thesehouseholdswereunique  0 intheirgeographicarea(blockfor100percentdataandblockgroupforsampledata)basedon    certaincharacteristics.Thedatafromthesehouseholdswereswappedwithdatafrompartnered    householdsthathadidenticalcharacteristicsonacertainsetofkeyvariablesbutwerefrom   differentgeographiclocations.Theswappingprocedurewasperformedindependentlyforthe x  100percentdataandthesampledata. ^  Tomaintaindataquality,therewasamaximumpercentofrecordsthatwereswappedforeach *z  stateforthe100percentdataandanothermaximumpercentforthesampledata.Presumably, ` thehighertherateofswapping,thegreatertheconfidentialityprotectionbutthelowerthedata F quality.However,thewaytheprocedureistargetedtorecordswithdisclosureriskandthe , choiceofvariablesthatarecontrolledonandthechoiceofvariablesthatarenotswappedalso  affecttheresultinglevelsofprotectionandquality.ThemaingoalwastoseeiftheCensus  Bureauwereabletostriketherightbalancebetweenprotectingconfidentialityandmaintaining  dataquality. t Toanswerquestionsondataquality,evaluatorscomparedtablesfromswappedversus @ unswappeddata,examinedthechangesincellvaluesduetotheswappingforcellsofdifferent &v sizes,andcomparedswappedandunswappedsampleestimatesof100percentdataitems.  \ Evaluatorsalsocomparedtheeffectsofswappingamongdifferentgeographiclevels.Toanswer B questionsondataprotection,evaluatorslookedathowoftentheCensusBureauwasabletoswap ( householdswithahighdisclosurerisk.Somecalculationswereperformedonall50states.For  themostdetailedanalysis,calculationswereperformedonthreestates(Oklahoma,  Massachusetts,andMississippi)forthe100percentdataandthreestates(WestVirginia,New   Jersey,andVermont)forthesampledata. p!  Forthe100percentdata,allrecordsweregivenachanceofbeingswapped.Theswappingwas <#" appliedconsistentlyineachstate.Recordswereassignedalevelofdisclosureriskfrom1to4 "$r# with4havingthemostdisclosurerisk.Theprocedureforassigningthelevelsofdisclosurerisk %X $ isCensusConfidential.Alllevel4recordswereswapped.Theperformanceonlevels3,2,and %>!% 1variedfromstatetostateandwasgenerallybetterforurbanstateswithadiversepopulation. &$"& Forthesampledata,allrecordsweregivensomechanceofbeingswapped.Asmallpercentof (#( householdswereswappedineachstate.Againrecordswereassignedalevelofdisclosurerisk. )$) Recordswerechosenforswappingbasedontheirlevelofdisclosureriskandtheabilitytopair l*%* recordswithhighlevelsofdisclosurerisk.Mostrecordsdeemedashavingadisclosurerisk R+&+ wereswapped. 8,', _ԇUsingvariablesthatarecommontoboththe100percentandsampledata,theCensusBureau  foundthattheconfidenceintervalabouttheswappedsampleestimatecoversthetrue100  percentvaluenearlyasoftenastheintervalabouttheunswappedestimate.Resultswerebetter | inurbanstateswithadiversepopulation. b Thedataswappingprocedurewascheckedforquality.Itwasconductedcorrectlyand . ~ consistently.Minimumbutnecessarychangesweremadetothedatainsuchawaythat  d maximizeddataquality.  J ThedisclosurelimitationmodelusedforCensus2000isusefulandtheCensusBureaushould    continuefutureresearchondisclosurelimitationtechniques.TheCensusBureaushouldinclude    confidentialityprotectionaspartoftheprocesswhenplanningacensus.    CategoryDPartnershipandMarketing  D  SummariesofthethreeevaluationsinthePartnershipandMarketingcategoryfollow. `  EvaluationD.1,PartnershipandMarketingProgramEvaluation (Wolteretal2002) , TheCensusBureaucontractedwiththeNationalOpinionResearchCentertoevaluatewhether  thePartnershipandMarketingProgramincreasedthepublicsawarenessofthecensusand  mailbackresponserates,especiallyamonghistoricallyundercountedpopulations.TheNational t OpinionResearchCenterimplementedabefore,during,andafterresearchdesignwiththree Z wavesofinterviewing.Wave1occurredinFall1999beforethelaunchoftheeducationphase @ oftheadvertisingprogramandbeforemostpartnershipactivitieshadcommenced;Wave2took &v placeinWinter2000beforethemailoutofcensusforms;andWave3beganinSpring2000  \ followingCensusDayandcontinuedduringthecensusNRFUoperations.Acrossthethree B wavesofdatacollection,theNationalOpinionResearchCentercompletedjustunder10,000 ( interviewsofAmericanhouseholds.Thesurveyssoughttointerviewthepersoninthe  householdwhoopensthemailortheonemostlikelytoopenandanswerthecensusform.  Theresearchdesignincorporatedrepresentativesamplesofseveralrace/ethnicitypopulations, p!  includingHispanics,nonHispanicBlacks,nonHispanicWhites,Asians,AmericanIndians,and V"! NativeHawaiians.Itenabledseparateanalysisandconclusionsabouttheeffectivenessofthe <#" PartnershipandMarketingProgramfortheserace/ethnicitypopulations,aswellasforthetotal "$r# population.Thedesignalsoincorporatedanexactmatchofthesurveyresponsestotheactual %X $ censusreturnsforthehouseholdsinterviewedinWaves2and3. %>!% OverallawarenessofcommunicationsaboutCensus2000increasedsignificantlyovertime.It ' #' wasgreateraftertheCensus2000PartnershipandMarketingProgramthanbeforetheonsetof (#( theprogram.AwarenessofcommunicationsaboutCensus2000increasedforallsixofthe )$) race/ethnicitypopulations,includinghistoricallyhardtoenumeratepopulationssuchas l*%* Hispanics,nonHispanicBlacks,andAmericanIndians.Itappearsthattheprogramwas R+&+ effectiveforalltargetedpopulationsinstimulatingawareness. 8,', ЇTheeffectsofcensusmarketingandpartnershipactivitiesareconfoundedwithoneanother.As  such,itwasimpossiblefortheevaluationtomeasuretheireffectsseparately.Thestudydid  examine,however,thepublic'srecallofeighteensourcesofcensuscommunications,eachof | whichexhibitedacombinationofadvertisingandpartnershipinfluences.Tostrengthenthe b analysis,evaluatorscombinedtheeighteensourcesintotwocompositemeasures:massmedia H andcommunitybasedcommunications.Massmediacommunicationsincludedtelevision, . ~ magazine,radio,newspaper,andbillboardads.Communitybasedcommunicationsincluded  d religiousgroups,communityorgovernmentorganizations,informalconversations,schoolsyou  J attended,schoolsyourchildrenattended,censusjobannouncements,conferenceexhibitbooths,  0 signsinsidebuildings,speeches,articles,theInternet,paycheckorutilitybill,andparticipation    onacompletecountcommittee.Resultsshowedsignificantevidencethatawarenessofboth    typesofcommunicationswasgreateraftertheCensus2000PartnershipandMarketingProgram   thanbeforetheonsetoftheprogram. x  Fourrace/ethnicitypopulations(nonHispanicBlacks,nonHispanicWhites,Asians,andNative D  Hawaiians)indicatedthattheyweremorelikelytoreturnthecensusformaftertheCensus2000 *z  PartnershipandMarketingProgramthanbeforeitsonset.Higherawarenessofcommunications ` aboutCensus2000correlateswithagreaterlikelihoodorintentionofreturningthecensusform F forfiveofthetargetedpopulations(Hispanics,nonHispanicBlacks,nonHispanicWhites, , Asians,andNativeHawaiians).Hispanicsshowthiseffecteventhoughtheirmeanintended  participationdidnotincreasefrombeforetoaftertheCensus2000PartnershipandMarketing  Program,suggestingthattheprogramhadlessimpactonthem.  Accordingtothedata,attitudestowardscensusconfidentialitydeclinedatthecloseofthe20th Z Century.FavorableattitudesstartedatalowlevelpriortoCensus2000andneverrecoveredto @ thelevelsreportedin1990.Ontheotherhand,respondentsviewsoftheimportanceof &v participatinginthecensusremainedquitestable:bothcensusesexhibitedsimilarlyfavorable  \ attitudesandneitherdisplayedatrendfromwavetowavewithinthecensusperiod.Finally,the B Census2000PartnershipandMarketingProgramseemstohaveachievedgreatersuccessthan ( comparableeffortsin1990tocreateafavorableattitudethatthecensuscannotbeusedagainst  you.  Inlightofthesefindings,therecommendationsare: p!   * #$%&'((xx * #$%&'(" i"   i23  0    Repeataprogramofmassmediaandcommunitybasedcommunicationsingeneralform, <#" content,andintensityforthe2010Census. i݌"$r#(#(# Ќ  " i"   i#23  0    Evaluatecurrentcommunicationschannels,withaneyetowardsoptimizingthe %>!% allocationofPartnershipandMarketingProgramresourcesamongthevariouschannels. i#P݌&$"&(#(# Ќ  " i"   i23  0    ReevaluatewhatpromotionalmessagesresonatebestwiththeAmericanpopulation, (#( overall,andwithtargetedrace/ethnicitypopulations. i݌)$)(#(# Ќ  " i"   i23  0    BuildonthesuccessoftheCensus2000PartnershipandMarketingProgramforthe R+&+ Blackpopulation. i݌8,',(#(# Ќ  " i"   i23  0    ReevaluatethecommunicationsapproachfortheHispanic,NativeHawaiian,non  EnglishspeakingAsians,andAmericanIndianpopulations. i>݌(#(# Ќ  " i"   id23  0    Conductaformalcostbenefitanalysis,attemptingtodemonstratethetradeoffsbetween b increasedexpendituresonPartnershipandMarketingProgramactivitiesandreduced H followupcosts. id݌. ~(#(# Ќ  " i"   i23  0    Useanexperimentaldesigntomeasuretheeffectivenessandbenefitofapartnershipand  J marketingprogram. i,݌ 0(#(# Ќ  EvaluationD.2,EvaluationoftheCensusinSchoolsProgram:MaterialsandDistribution     (MacroInternational2002)   TheCensusinSchoolsProgramaimedtoraiseawarenessofCensus2000.Theprogramoffered ^  teachingmaterialsthatprovidedinformationonthepurposesandmethodsofthecensusandthat D  soughttoengenderaninterestinthecensus.Theprogramwasparticularlytargetedatschoolsin *z  hardtoenumerateareas.Thisevaluationwasintendedtomeasuretheeffectivenessofthe ` approachfordisseminatingCensusinSchoolsProgrammaterialsanduseofandsatisfaction F withthematerialsamongteachers.ItwasnotintendedtomeasuretheimpactoftheCensusin , SchoolsProgrammaterialsonchildrenortheirparentsoronensuringahighrateofparticipation  inCensus2000.  Allelementaryschoolteachersandallsecondarymathorsocialstudiesteachersinhardto t enumerateareasweresentaninvitationalpacket.Thisinvitationalpacketconsistedofan Z informationalletterandanorderform.Thispacketprovidedteacherswiththeopportunityto @ orderCensusinSchoolsProgrammaterials,whichincludedaTeachingGuide,lessonplans,and &v aGiantU.S.map.Principals,otherthanthoseinhardtoenumerateareas,administrators,and  \ curriculumcoordinatorsalsoreceivedaninvitationalpacket.Additionally,allelementaryschool B teachersandmiddleschoolsocialstudiesteachersweresentTakeHomematerialsforstudents ( tolearnaboutthecensusandsharewiththeirparentsathome,therebyhavingthepotentialto  reacheachkindergartentoeighthgradestudentinthecountry.  ORC/MacroInternationalconductedtheevaluationoftheCensusinSchoolsProgrambasedona p!  surveytheyfieldedinSpring2000.Theymailedtoastratifiedrandomsampleof4,000teachers V"! selectedfromallprimaryandsecondaryteachersinthe50states,DC,andPuertoRicoandfrom <#" teachersorderingtheCensusinSchoolsmaterials.Oftheteachersselectedforthesurvey,1,101 "$r# responded. %X $ Approximately56percentofallteachersheardoftheCensusinSchoolsProgram.Thesingle &$"& mostimportantconduitforinformationabouttheprogramwastheinvitationalpacket.Fifty ' #' fourpercentofteachersheardabouttheprogramthroughinvitationalpackets.Approximately23 (#( percentheardaboutitfromtheirprincipal.Overall,63percentofallteacherswhoheardabout )$) theCensusinSchoolsProgramreceivedatleastonecomponentoftheCensusinSchools l*%* Programmaterials.Thirtyninepercentofteacherswhoreceivedthematerialsacquiredthem R+&+ fromtheirprincipals.About23percentoftheteacherswhoreceivedthematerialsorderedthem 8,', inresponsetotheinvitationalpacket.TeacherswhoheardabouttheCensusinSchoolsProgram  fromtheirprincipalratherthanhearingofitthroughothersources,weremorelikelytoactually  receivethematerials. | TheGiantU.S.Mapwaspopular.Ofthe85percentofteacherswhoreceivedthemap,92 H percentuseditinclassroomactivities.Ofthe33percentofteacherswhoreceivedtheTake . ~ Homematerials,about79percentsentthemhomewiththeirstudents.  d About39percentofallteachersinthe50states,DC,andPuertoRicotaughtinhardto  0 enumerateareas.OfthoseinhardtoenumerateareaswhoheardoftheCensusinSchools    Program,almost61percentreceivedatleastonecomponentoftheCensusinSchoolsProgram    materials.SixtyfourpercentofteachersinhardtoenumerateareaswhoheardabouttheCensus   inSchoolsProgramthroughtheinvitationalpacketreceivedatleastonecomponentofthe x  CensusinSchoolsmaterials.Teachersinhardtoenumerateareasweremorelikelytohave ^  orderedthematerialsthemselves(34percent)thanteachersinotherareas(16percent). D  Fiftythreeteachersprovidedanswerstoanopenendedquestionaboutwhytheydidnotsend ` TakeHomematerialshome.Responsesincludedthatteachersneedmoreleadtimetoexamine F theCensusinSchoolsProgrammaterialsandincorporatethemintotheircurricula.Some , teachersfoundtheCensusinSchoolsProgrammaterialstoodifficultfortheirstudentsand  othersthoughttheyweretooelementarySometeachersreceivedtheCensusinSchoolsProgram  materialsinthewronglanguageortargetinganinappropriateagegroup.  Inexaminingtheresultsofthisevaluation,somethemesappeared: Z " i"   i_23  0    PrincipalswereanimportantconduitfortransferringinformationaboutCensusin &v SchoolsProgrammaterialsaswellasfororderingthematerials. i_݌ \(#(# Ќ  " i"   i23  0    Itseemsthattheinvitationalpacketdidnotdrawtheattentionofmanyteacherstowhom ( itwassent.Teachersreceivemanyitemsintheirmailboxesandtheinvitationalpackets  didnotappeartostandoutfromothermaterialssenttoteachers. i݌(#(# Ќ  " i "   i23  0    ForthoseusingtheCensusinSchoolsProgrammaterials,satisfactionwashigh. i݌p! (#(# Ќ  Inviewofthesefindings,therecommendationsare: <#" " i "   i23  0    Conductresearch/testingtounderstandhowandwhenteachersreacttovarioustypesof %X $ mailingsandtobetterunderstandhowteacherscanbereached. i݌%>!%(#(# Ќ  " i "   iA23  0    Designthematerialstobettermeettheneedsoftheteachers. iAn݌' #'(#(# Ќ  " i "   i723  0    UseprincipalstotransmittheCensusinSchoolsProgrammaterialstoteachers. i7d݌)$)(#(# Ќ   R+&+  EvaluationD.3,ReportofSurveyofPartners (Westat2001)  ThemissionofthePartnershipProgramwastodevelopanaggressiveandcomprehensive | programthatincorporatedtheeffortsandresourcesofgovernmentunits,communitybased b organizations,religiousgroups,andbusinessestoassisttheCensusBureauinconductingan H efficient,accuratecensus.Theprimarygoalsoftheprogramweretoincreasemailresponse . ~ rates,reducethedifferentialundercounts,andcommunicateaconsistentCensus2000message.  d Toachievethesegoals,theCensusBureauformedpartnershipswithstate,local,andtribal  J governments,nongovernmentalorganizations,communitygroups,themedia,andprivatesector  0 businesses.TheCensus2000PartnershipProgramalsoincluded690partnershipstaffat    headquartersandacrossall12regions.    Asurveybasedstudywasconductedtoevaluatetheprogram'seffectivenessfromthepartners' x  viewpoint.Amodeloforganizationalrelationshipswasusedasanorganizingframeworkforthe ^  evaluation.Thecomponentsofthemodeladdressedbythesurveywere: D  " i "   i23  0    BenefitspartnersexpectedtoachievefromtheirpartnershipswiththeCensusBureau. iϫ݌`(#(# Ќ  " i"   i23  0    CensusBureaucontributionstothepartnership(thewidevarietyofmaterialsCensus , providedtoparticipatingorganizations). i۬݌(#(# Ќ  " i"   i23  0    Partnercontributionstothepartnership(activitiespartnersconductedtopublicizeand  increaseawarenessofthecensus,togettheirtargetpopulationscounted,andtoassist t withCensusBureauoperationsandinitiatives,financialcontributions,andinkind Z contributionspartnersmadetosupportandpromoteCensus2000efforts). i$݌@(#(# Ќ  " i"   i823  0    StructuresandprocessesthatexistedbetweentheCensusBureauandpartnersto  \ accomplishpartnershipgoals. i8e݌B(#(# Ќ  CensusBureaustaffdrewastratifiedrandomsampleof15,803fromaframeofpartnersthat  wereenteredintheContactProfileUsageandManagementSystematthetimethesamplewas  drawn.DatawerecollectedoverasixmonthperiodbybothmailandComputerAssisted   TelephoneInterviewing(CATI)methods.Thesurveyachieveda67.9percentresponserate. p!  Seventyto81percentofpartnersrespondedthattheyplaced Moderateemphasisor Alotof <#" emphasisoneachoffivePartnershipProgramgoals.Partnersexpectedbenefitsof "$r# participationwerealignedwiththeCensusBureausgoalsfortheprogram.Fromthepartners' %X $ view,contributionstheCensusBureaumadewerehighlyvalued.Themajorityofpartners %>!% (rangingfrom71.3percentto88.0percent)thatusedeachofthe18typesofmaterialsratedthe &$"& materialas"ModeratelyHelpful"orVeryHelpful."NonEnglishmaterialswereusedbymore ' #' than90percentofallorganizationsthatreceivedthemandthesematerialswerealsoratedas (#( "ModeratelyHelpful"or"VeryHelpful"bymorethan80percentofpartnersthatusedthem. )$) Seventypercentofrespondentorganizationsreportedthattheyconductedoneormoreactivities. R+&+ Meanratingsforallactivitieswereabovethe3.1levelofthefourpointscale(1=Notatall 8,', Helpfulto4=VeryHelpful),indicatingthatacrossallpartners,everyactivitywasconsidered  tobeatleastModeratelyHelpfulinachievingPartnershipProgramgoals.Relativelyfewof  thepartnersrespondedthattheymadeanytypeoffinancialcontributionstothepartnership. | Resultsindicatedthatpartnerscontributedmoreintermsofresources(e.g.,stafftime,space, b materials,etc.)ratherthanspendingorganizationalfunds. H Themajorityofpartners(70percent)reportedthatCensusPartnershipSpecialistshelpedthem  d promoteCensus2000.MorethanhalfofthepartnersreportedthatthedirectCensussupportand  J Censusparticipationintheiractivitieswashelpful.Overall,partnersweresatisfiedwiththe  0 processinplacetofurnishthemwithPartnershipmaterials.Amajorityofpartnersindicatedthat    thePartnershipProgramhelpedthemtoreachtheirgoalsforparticipating,moresoforgoalsof    reachingandeducatingthetargetpopulation(67percentand72percent,respectively)thanfor   minimizingthetargetpopulation'sfearofprovidinginformationtothegovernment(60percent). x  Ofpartnersexpressinganopinion(79percentofallpartners),84percentwerepositiveabout D  theirintenttoparticipateaspartnersagain.Thisresultsuggeststheoverallsuccessofthe *z  program. ` Recommendationsforpracticesthatshouldstaythesame: , " i"   ij23  0    Continuetodefinecommongoalsthatpartnersperceiveasbenefits. ij݌(#(# Ќ  " i"   ie23  0    Continueuseofthevarietyofmaterialsforeducationandawareness.Continuetomake t useofthespecificmaterialsthatwereratedmostusedandmosthelpful.Continueto Z developanduselanguageappropriatematerials. ie݌@(#(# Ќ  " i"   i#23  0    Encouragefuturepartnerstoconductthetypesofactivitiesthatpartnersconsidered  \ successfulduringCensus2000. i#P݌B(#(# Ќ  " i"   ic23  0    ContinuetoprovideliaisonsupporttopartnersthroughPartnershipSpecialists. ic݌(#(# Ќ  " i"   ij23  0    ContinuetoprovidedirectCensussupportforpartneractivitiesandCensusparticipation   inthoseactivities. ij݌p! (#(# Ќ  Recommendationsforpracticesthatshouldchange: <#" " i"   i23  0    Makepartnerbenefitsmoreexplicit. i݌%X $(#(# Ќ  " i"   i23  0    Reevaluatetheuseofspecificmaterialsthatwereratedleastusedandleasthelpful. i݌&$"&(#(# Ќ  " i"   i23  0    Improvetheprocessforfurnishingmaterialstopartners. i݌(#((#(# Ќ  " i"   i23  0    Incorporatequalitativefeedbackobtainedfrompartnersintoanyplansforimprovement l*%* forthenextcensuscycle. i݌R+&+(#(# Ќ   8,', Ї CategoryESpecialPlacesandGroupQuarters   SummariesofthethreeevaluationsintheSpecialPlacesandGroupQuarterscategoryfollow. |  EvaluationE.1.b,EvaluationoftheFacilityQuestionnaire(ComputerAssistedTelephone H InterviewingandPersonalVisit) (Stevens2003) . ~ Thisevaluationcomparedtheclassificationofspecialplaces,alsoknownasspecialplacetype  J coding,collectedatthreedifferenttimesduringCensus2000.Itspecificallyincluded  0 comparisonsofthespecialplacetypecoderecordedduringtheSpecialPlaceFacility    Questionnaireoperation,Reinterview,andtheSpecialPlaceAdvanceVisitoperation.Special    placetypecodingwasconductedtodeterminethepaththeremainderoftheinterviewwould   followintheSpecialPlaceFacilityQuestionnaireandSpecialPlaceAdvancedVisitoperations. x  Inaddition,subsequentoperationsofdatacollectionweredependentonit.Thisevaluationwas ^  designedtoanswertwoquestionsrelatedtospecialplacetypecodingwhichhelpdescribethe D  consistencyofthespecialplacetypecodingacrossthethreeoperationslistedabove. *z  Notethat,althoughtheSpecialPlaceFacilityQuestionnaireandSpecialPlaceAdvanceVisit F operationsrecordedspecialplacetypecodes,theprimarypurposeoftheseoperationswasto , collectgroupquarterstypecodes.Thisevaluationfocusedonspecialplacetypecodes.The  conclusionsfromthisevaluationshouldnotbeappliedtogroupquarterstypecodes.The2010  Censusplanistoemphasizegroupquarterstypecodesratherthanspecialplacetypecodes.  Therefore,resultsfromthisevaluationmaynotbeinfluentialtothe2010design. t About25percentofthespecialplacecodeschangedfromtheSpecialPlaceFacility @ QuestionnairetotheReinterviewand38percentofthespecialplacetypecodeschangedfrom &v theReinterviewtotheSpecialPlaceAdvanceVisit.OmittingtheReinterviewoperation,25  \ percentofthespecialplacetypecodeschangedfromtheSpecialPlaceFacilityQuestionnaireto B theSpecialPlaceAdvanceVisit.However,ofthe25percentthatchangedfromtheSpecial ( PlaceFacilityQuestionnairetotheSpecialPlaceAdvanceVisit,mostaredifferentbecausethe  specialplacewasdeletedfromthecensusandnotbecauseofachangeinthecodingofthe  specialplace.   Discrepanciesinthespecialplacetypecodediffersomewhatbytypeofspecialplace. V"! Specifically,large/complexspecialplacesweremorelikelytohaveadiscrepancyinthespecial <#" placetypecodethanotherspecialplaces.However,thisdifferencemaynotbeofpractical "$r# significance. %X $ Recommendationsincludethatfutureevaluationsbasethecomparisononthegroupquarters &$"& typecodeinsteadof,orinadditionto,thespecialplacetypecode.Agroupquarterstypecode ' #' comparisonismoreappropriatesinceitistheclassificationbywhichdataaretabulatedincensus (#( products. )$) _  R+&+  EvaluationE.5,Revision1,GroupQuartersEnumeration (Jonas2003b)  TheGroupQuarterspopulationconsistsofallpersonswhodonotliveinhousingunitssuchas | singlefamilyhouses,apartments,andmobilehomes,butratheringroupsituationssuchas b collegedormitories,nursinghomes,militarybarracks,prisons,juvenileinstitutions,migrant H workerdormitories,convents,andgrouphomes.ASpecialPlaceisanadministrativeentity . ~ containingoneormoreGroupQuarters.TheGroupQuartersarewherepeoplesleep.For  d instance,auniversityisaSpecialPlaceandeachdormitoryisaGroupQuarters.Thisevaluation  J wasdesignedtodocumentthecountsofSpecialPlaces,GroupQuarters,andGroupQuarters  0 populationandotheroperationalaspectsoftheGroupQuartersenumeration.    GroupQuartersenumerationsucceededinitsunderlyingmissionofgainingafundamentally   accuratecountoftheGroupQuarterspopulation.TheGroupQuartersuniversewashometo7.8 x  millionpeoplein2000.Thesepeoplewereenumeratedin192,286GroupQuartersin100,358 ^  SpecialPlaces.Universities,militarybases,andcorrectionalinstitutionswerethelargestspecial D  places,asmeasuredbybothsizeofpopulationandnumberofGroupQuartersperSpecialPlace. *z  SpecialPlaceswith1,000ormorepeopleweremadeupalmostentirelyofpersonsincolleges, ` prisons,andmilitarybases(99percentofthatpopulationwereinthesetypesofplaces). F Overall,fortypercentofSpecialPlaceshadlessthantenresidentsand61percenthadlessthan  25residents.TheseweremostlygrouphomesandServiceBasedEnumeration(SBE)sitesor   otherSpecialPlaces.The40percentofSpecialPlaceswithlessthantenresidentshadonly  2.3percentoftheGroupQuarterspopulation. t MoreGroupQuartersquestionnaireswerefilledoutfromadministrativedatathanbyanyother @ method,withnursinghomes,hospitals,grouphomes,andcorrectionalinstitutionsusing &v administrativedatamostfrequently.  \ Over200,000GroupQuarterspersonrecords(2.6percentofallGroupQuarterspersonrecords) ( hadallcharacteristicsimputed.Over55,000ofthesepersonswereenumeratedon  questionnairesthatcouldnotbeprocessedbecausethehandtranscribedGroupQuartersIDwas  eithernotenteredontheformbyanenumeratororwasanerroneousIDthatcouldnotbe   identifiedwithaGroupQuarters. p!  Anestimated4.4percentofpersonsinGroupHomesorreligiousGroupQuarterswerecounted <#" twice;oncebytheGroupQuartersenumerationprocessandagainonahouseholdquestionnaire "$r# returnedbymail.SchedulingconstraintsdidnotallowCensusaddressestoberemovedfromthe %X $ housingunitaddressmaillistiftheywerealsofoundtobeincludedasaGroupQuartersaddress. %>!% Assuch,someaddresseswereidentifiedasduplicateaddressesbutenumeratedtwice.This &$"& occurredprimarilyamongGroupHomesandreligiousGroupQuarters. ' #' Nearly150,000housingunitswereidentifiedbyGroupQuartersenumeratorsatGroupQuarters )$) andattransientlocationssuchasrecreationalvehicleparks.Thesehousingunitscontributed l*%* over260,000personstothecensus. R+&+  8,', RecommendationsincludeanumberofchangestoGroupQuartersenumerationfor2010:   * #$%&'((xx* #$%&'(" i"   i23  0    Useavailabletechnologytotrackindividualquestionnairesfromenumerationtodata | capture. i݌b(#(# Ќ  " i"   i23  0    ImprovetheaddresslistcreationprocessbygatheringdataonSpecialPlacesfromWeb . ~ basedsources,tailoraddresslistcreationandenumerationstrategiestoeachmajor  d categoryofGroupQuarters,andgivelargeSpecialPlacestheoptionofprovidingGroup  J Quartersdatabyelectronicorprintedrecordsratherthanbytelephoneandinperson  0 interviews. i݌  (#(# Ќ  " i"   iX23  0    ReduceduplicationbetweentheaddressfilesforGroupQuartersandhousingunits. iX݌ (#(# Ќ  " i"   ia23  0    Bepreparedforuseofadministrativedatainenumeration. ia݌^ (#(# Ќ  EvaluationE.6,ServiceBasedEnumeration (McNally2002) *z  ThegoalofSBEwastoprovidepeopleexperiencinghomelessnessanopportunitytobeincluded F inthecensus.TheCensusBureaudevelopedaspecializedoperationtoenumerateselected , servicelocationsthatservepeopleexperiencinghomelessness.Itisimportanttonotethatthe  Census2000countoftheSBEpopulationdoesnotrepresentacompletecountofpeople  experiencinghomelessness.  BetweenMarch27andMarch29,2000,theCensusBureauenumeratedpeopleatemergency Z shelters,soupkitchens,regularlyscheduledmobilefoodvans,andtargetednonshelteredoutdoor @ locations.PeopleonBeCountedFormswhomarkedthe NoAddressonApril1,2000"boxor &v indicatedtheywerehomelessintheaddresssectionalsowereincludedintheSBEuniverse.  \ Therewere14,817SBEsitesinCensus2000.Morethanhalf(51percent)ofthelocationswere ( shelters.Therewereatotalof258,728personrecordsdatacapturedfromshelters,soup  kitchens,regularlyscheduledmobilefoodvans,andtargetednonshelteredoutdoorlocations.  Mostofthedatacapturedpersonrecords(90percent)werefromshelters,soupkitchensand   regularlyscheduledmobilefoodvans. p!  TheSBEoperationappearstobeasuccessfulmethodofincludinginthecensuspeoplewhoare <#" experiencinghomelessness.Atotalof283,898peopleweretabulatedinCensus2000asaresult "$r# oftheSBEoperation.Thistotalincludes9,963personsimputedintotheCensus2000 %X $ population. %>!% Therewasatotalof258,728questionnairescapturedfromtheSBEprogram.Almostall(99 ' #' percent)ofthedatacapturedpersonrecordshadatleasttwoormoredatacharacteristics(name, (#( sex,ageand/ordateofbirth,Hispanicorigin,andrace). )$)   R+&+ Approximately87percentofthedatacapturedpersonscontainedenoughinformationforthe  questionnairetobeincludedintheunduplicationprocess.Thatis,thequestionnairehadafirst  andlastnamewithcombinedfieldscontainingatleastthreealphabeticcharactersandatleast | twopersoncharacteristics,oneofwhichwasdateofbirthorage.TheCensusBureauwasable b tomatchandunduplicate16,787personrecordsduringdataprocessing.However,itwas H discoveredthat2,410oftheserecordswereerroneouslyidentifiedasaduplicaterecord. . ~ Atotalof38,415peoplecompletedaBeCountedFormandmarkedthe NoAddresson  J April1,2000"boxonthatformorindicatedtheywerehomelessintheaddresssection.Of  0 these,theCensusBureauwasabletomatchandunduplicate3percenttopeopleenumerated    duringtheSBEoperation.TheBeCountedprogramadded35,121personswithoutausual    addresstoCensus2000.Mostofthese(31,994)weretabulatedwiththepopulationforSBE   sites.TheremainderweretabulatedwiththeGroupQuarterspopulation. x  Basedontheresultsofthisevaluation,theCensusBureaushouldcontinuesupportingSBEfor D  the2010Census. *z   CategoryFAddressListDevelopment  , SummariesofthetenevaluationsintheAddressListDevelopmentcategoryfollow.   EvaluationF.2,TheAddressListingOperationandItsImpactontheMAF (Ruhnke2002) t TheevaluationoftheAddressListingoperationforCensus2000examinedtheoperation's @ impactoncreatingtheMAFforcertainareasofthecountry.TheCensusBureauconductedthe &v AddressListingoperationfromJuly1998toMay1999andusedtheresultstocreatetheinitial  \ addresslistforareasthatwouldbeenumeratedusingU/LmethodologyduringCensus2000.In B theAddressListingoperation,censusenumeratorscanvasseddoortodoortoidentifythe ( mailingaddressandphysicallocationofaddressesinareaswheretheCensusBureaubelieved  thatproblemswerelikelywithdevelopinganaccuratemailinglistanddeliveringcensus  questionnairesthroughthemail.Theenumeratorsalsolocatedeachhousingunitwithamap   spotonablockmapandcollectedanoccupantnameandtelephonenumber,whenpossible. p!  Thisevaluationlookedatthenumber,geographiclocation,characteristics,andqualityof V"! addressesl_isted_ԀduringtheAddressListingoperation. <#" Stateside,about22millionhousingunitswerelistedintheAddressListingoperation.An %X $ additional1.4millionaddresseswerelistedinPuertoRico.AllofPuertoRicowascanvassed %>!% duringtheAddressListingoperationandwasenumeratedusingU/Lmethodology. &$"& DespiteAddressListingoccurringinmostlyruralareasoftheU.S.,over73percentoftheunits (#( hadcompletecitystyle(housenumber,streetname)addresses.About14percentoftheunits )$) hadincompleteornoaddressinformation,butlocationdescriptionsoftheunitswererecorded l*%* forover95percentofthoseunits.Bothcitystyleaddressinformationandlocationdescriptions R+&+ enableenumeratorstolocatetheunitsonthegroundwhentheydeliverthecensusformsduring 8,', _U/Landothercensusfieldoperations.Thepresenceofamapspot,auniqueidentifierfora  housingunitonacensusmapwithinablock,isalsocrucialwhentryingtolocateaunitinrural  areas.Over99percentoftheAddressListingaddshadmapspots. | AddresseseligiblefortheDMAFincludedthosethatrepresentpotentialresidentialhousingunits H thatarecodedtocensusblocksandhavemapspots.Over99percentoftheAddressListing . ~ addsweredeliveredtotheDMAFandapproximately94percentofallAddressListingadds  d wereincludedinthefinalCensus2000counts.  J Inareaswheremostmailingaddressesarecitystyle(forexample,101MainStreet),theCensus    BureaucreatedtheMAFbycombiningaddressesfromthe1990CensusAddressControlFile    withaddressesintheUSPSDeliverySequenceFile(DSF).Approximately43percentof   addressesaddedinAddressListingmatchedtoaddressesthatwereidentifiedasresidentialonor x  beforetheSeptember1998USPSDSF.About280,000blocksinU/Lareashadalloftheir ^  addressesmatchtotheDSF.Thisisabout14percentofallblocksinwhichtherewasatleast D  oneunitlistedduringtheAddressListingoperation. *z  Listerswereallowedtwotelephonecallbackstocollectmailingaddressinformationduringthe F AddressListingoperation.Therewerethreeadditionalpersonalvisitcallbacksusedtoobtain , addressinformationin36oftheapproximately3000countiesinwhichAddressListingwas  done.The36countieswerethesitesofthe1999ACS.Theadditionalcallbacksweremadeto  maximizemailresponseinthatsurvey.Itappearsthattheadditionalcallbacksmayhave  contributedtothesuccessofobtainingadditionaladdressinformation,althoughnotinany t significantmanner. Z Recommendationsresultingfromthisevaluationinclude: &v  * #$%&'((xx* #$%&'(" i"   i23  0    ReassessthemethodologyofdelineatingMailout/MailbackversusU/Lareas.Itmaybe B reasonableinsomeCensus2000U/LenumerationareastousetheDSFasanaddresslist ( buildingtool. i݌(#(# Ќ  " i"   iS23  0    Sincetheimpactoftheadditionalcallbacksonobtainingmailingaddressinformation   appearssmallandthenecessarycostdatawerenotavailabletodoaneffectivecost p!  comparison,additionalcallbacksforafutureAddressListingoperationarenot V"! recommendedatthistime. iS݌<#"(#(# Ќ  EvaluationF.3,EvaluationoftheLocalUpdateofCensusAddresses98(LUCA98) (Owens %X $ 2003) %>!% TheCensusBureauconductedtheCensus2000LUCA98programinmailout/mailbackareas ' #' fromMay1998toJune2000.TheCensusBureauinvitedlocalandtribalgovernmentsto (#( participate.Thosewhoparticipatedweresentlistsofhousingunitsinthecensusblocksintheir )$) area.TheaddresslistfortheLUCA98programincludedaddressesfromvariousMAFsources, l*%* includingthe1990AddressControlFile,twoUSPSDSFdeliveries,andtheBlockCanvassing R+&+ operation.Therewereapproximately81.5millionaddressesfromthesesourcesontheMAF 8,', thatwereeligibleforreviewintheLUCA98program.Governmentsupdatedthelistsby  adding,deleting,orcorrectingaddresses.TheCensusBureauthenverifiedmostofthose  updates.ThisevaluationdocumentedtheresultsoftheLUCA98program. | Therewere17,424governmentalunitseligibletoparticipateintheLUCA98program.Atotal H of9,263governmentsparticipated.Thehousingunitsinthesejurisdictionsgeographically . ~ coveredapproximately92percentofthehousingunitsinareaseligibleforLUCA98.Although  d abouthalfofalleligiblegovernmentsparticipated,alittlemorethanathirdofeligible  J governmentsprovidedanyupdatesintheformofadds,deletes,orcorrections.TheCensus  0 Bureaushouldinvestigatewaystoincreasegovernmentparticipation,especiallyfocusingon    waystoaidthegovernmentalunitinprovidingupdatesoncetheyhaveagreedtoparticipate.In    general,smallergovernments(asdeterminedbythenumberofhousingunitsinthe   governmentsjurisdictionin1990)hadlowerparticipationratesthanlargerones.Governments x  mayhavenotparticipatedbecausetheydidnothaveenoughresourcestodothetaskorthey ^  knewthatlargergovernmentsintheirareawerealreadyupdatingaddressesfortheCensus D  Bureau. *z  LUCA98participantsreviewedaddresslistsandaddedaddressesforresidentialunitsintheir F jurisdictionthattheybelieveddidnotexistontheirreviewmaterials.Theyadded5,302,094 , addressestotheMAF,whichrepresentsa6.5percentincreaseinhousingunitsin  mailout/mailbackenumerationareas.Approximately95percentofLUCA98participantadds  wereincludedontheinitialcensusaddresslist.Manywereaddedtotheinitiallistas   provisionaladds,tobeverifiedafterthefirstcensusmailing.Approximately58percentof t addswereconfirmedtoexistasresidentialaddressesintheBlockCanvassingoperationorthe Z LUCAFieldVerificationoperation.About58percentofaddswereinthefinalcensushousing @ unitinventory. &v TheLUCA98participantsdeleted(ordeclarednonresidential)anyaddressontheiraddresslist B thattheybelieveddidnotexistintheirjurisdictionasaresidentialunit.Theydeleted490,613 ( addressesfromtheMAF.LUCA98participantscorrected2,762,050addresses.Thecorrections  includedgeographicaswellasaddressinformation.  LUCA98participatinggovernmentsappealed313,853addresses.Atotalof303,410ofthose p!  addresseswereaddedtotheMAFafterapprovalbytheCensusAddressListAppealsOfficethat V"! wassetupbytheOfficeofManagementandBudget.Therewere141,580appealaddressesthat <#" wereincludedonthefinalCensusaddresslist. "$r# TheparticipantsoftheLUCA98programcontributedtotheaddresslistinmanyareas. %>!% AlthoughtheupdateshadalargeimpactontheupdateoftheMAFforCensus2000,thetiming &$"& oftheprogramwithotherCensus2000addressupdatingoperationsintroducedsomecomplexity ' #' indeterminingthetrueimpactofupdatestothefinalcensusresults.However,about505,530 (#( addressesinthefinalcensuswereprovidedbyLUCA98participantsandmaynothavebeen )$) providedbyanyothercensusoperation. l*%*  8,', TounderstandthetrueimpactofLUCAinthefuture,theCensusBureaushouldallowsufficient  timeforthecompletionofgovernmentupdatespriortoBlockCanvassingactivities.Thiswould  reducethecomplexityofprocessing,aswellaseliminatetheneedforanotheroperationto | validateupdates. b  EvaluationF.5,BlockCanvassingOperation (Burcham2002) . ~ TheBlockCanvassingoperationwasoneofthelargestoperationstheCensusBureauconducted  J toupdatetheMAFinpreparationforCensus2000.Itoccurredinthewinter/springof1999.  0 Theoperationrequiredfieldlisterstoconducta100percentcanvassofresidentialaddressesin    areascontainingpredominantlycitystyleaddresses.ResultsfromBlockCanvassingwereused    toassigneachhousingunittooneofsixbasicactioncodecategories:Verify,Add,Delete,   AddressCorrected,GeographicCorrections,andAddandVerifyB  d(6,page110,R1;67,page7,R1)dC  . x  ThisevaluationquantifiedtheimpactoftheCensus2000BlockCanvassingOperationonthe D  MAFbyprofilingthehousingunitsthatBlockCanvassingaddedto,deletedfrom,andcorrected *z  ontheMAF.ThisevaluationdidnotprovideathoroughcomparisonofBlockCanvassing ` resultstofinalcensusresultsortotheMAFbuildingprocessasawhole. F BlockCanvassinglistersadded6,389,271addressestotheirlistingpages.About95percentof  theaddedunitshadcitystyleaddresses.Basedonpreliminaryresults,BlockCanvassing  appearstohavehadahighlevelofgeocodingaccuracy.Over94percentoftheaddsshoweda  BlockCanvassingblockcodeequaltotheofficialblockcodeontheMAF.Around29percent t ofaddressesaddedbyBlockCanvassingactuallywereontheMAFbeforeBlockCanvassing Z occurredbutwereeitherungeocodeduntilBlockCanvassinggeocodedthem,movedtodifferent @ blocksbyBlockCanvassing,orconsiderednonresidentialuntilBlockCanvassingdetermined &v thattheywereresidentialunits.  \ BlockCanvassinglistersdeleted5,146,320addressesfromtheirlistingpages.Theoriginal ( sourceofanaddressisthefirstsourcethataddedtheaddresstotheMAF.Ingeneral,Block  Canvassingdeletedalargerproportionofaddressesthathadaneweroriginalsourcethan  addresseswithanolderoriginalsource.Forexample,the1990AddressControlFileshoweda   lowerpercentageofdeletesthantheNovember97DSF. p!  Around78percentoftheaddedunitswerevalidhousingunitsinCensus2000andalmost24 <#" percentofthedeletedaddresseswerelaterenumeratedashousingunitsinthecensus.About96 "$r# percentofaddressescodedasexistingbyBlockCanvassingendedupasvalidhousingunitsin %X $ thecensus.Also,96percentofalladdressessenttoBlockCanvassingtobeverifiedshowed &$"& consistentresultsbetweenBlockCanvassingandthecensus. ' #' BlockCanvassingprovidedalargenumberofupdatestotheMAF.BlockCanvassingnotonly )$) improvedthecoverageofaddressesontheMAF,butalsoimprovedthegeocodingofaddresses l*%* ontheMAF.BlockCanvassingplayedasignificantpartincorrectingunitdesignationsinmulti R+&+ unitbasicstreetaddresses.IftheBlockCanvassinglistershadnotcheckedindividual_addresses 8,', withinmultiunits,butonlyverifiedthenumberofunitsatthemultiunits,theMAFwouldnot  havethisaddedimprovement.  ArelativelylargenumberofBlockCanvassingaddsanddeletesturnedouttobeinconsistent b withfinalcensusresults.However,theconsistencybetweenBlockCanvassingandthecensus, H asawhole,appearstoberelativelygood. . ~ Recommendationsresultingfromthisevaluationinclude:  J " i"   i623  0    Continuetoexplorethepossibilityoftargetingareaswithcertaincharacteristicsas    priorityareasforupdatingthefile.Theclusteringresultsinthisevaluationshouldbea    firststeptoshowinghowtotargetareasforMAFupdates. i67݌ (#(# Ќ  " i"   i823  0    Makeadditionaleffortstoseeifqualityreviewprogramscanreduceinconsistenciesfor ^  addedanddeletedaddresses. i88݌D (#(# Ќ  EvaluationF.6,EvaluationoftheLocalUpdateofCensusAddresses99(LUCA99) (Owens ` 2002) F TheCensusBureauconductedtheCensus2000LUCA99programinU/Land  Update/Enumerate(U/E)areasfromJanuary1999toJune2000.Thisevaluationdocumented  theresultsoftheLUCA99operations.  Therewere30,375functioninggovernmentalunitseligibletoparticipateintheLUCA99 Z program.Atotalof10,925governmentsparticipatedandtheycoveredapproximately68percent @ ofthehousingunitsineligibleareas.Althoughabout36percentofalleligiblegovernments &v participated,only17percentofeligiblegovernmentsprovidedanyupdatesintheformofadds,  \ deletes,orcorrections. B Therewereapproximately23,227,788addressesfromAddressListing(intheU.S.andPuerto  Rico)thatweregeocodedwithamapspotandeligibleforreviewintheLUCA99program.The  CensusBureausent2,186,765addressesoutforreviewtoparticipatinggovernmentsinthe   statesideLUCA99Recanvassoperation.Fieldrepresentativesverifiedthatabout76percentof p!  themexistedasresidentialunits.Theydeletedapproximately6percentoftheaddressesand V"! determinedthatlessthantwotenthsofapercentwerenonresidential.Theymadecorrectionsto <#" theremaining18percentofaddressesontheirlists. "$r# TheCensusBureausentatotalof35,563addressesoutforreviewinPuertoRico.Field %>!% representativesverifiedthatabout93percentofthemexistedasresidentialunits.Theydeleted &$"& approximately7percentanddeterminedthatlessthanonetenthofapercentwerenonresidential. ' #' TherewerenocorrectedaddressesinPuertoRico. (#( FieldrepresentativesfortheLUCA99Recanvassoperationupdatedtheaddresslistandadded l*%* anyunitthatexistedasaresidentialunitintheblockthatwasnotalreadyonthelist.They R+&+ added328,174addresses,whichrepresentsa15percentincreaseinhousingunitsinU/L 8,', enumerationareasintheU.S.(excludingPuertoRico)thatwererecanvassed.Field  representativesadded9,874addressesinPuertoRico,whichrepresentsabouta28percent  increaseinhousingunitsinareasthatwererecanvassed.Approximately99.5percentofLUCA | 99RecanvassaddsintheU.S.andPuertoRicowereincludedontheinitialcensusaddresslist. b About85.2percentofthoseaddswereinthefinalcensushousingunitinventory. H LUCA99Recanvassfieldrepresentativesdeleted(ordeclarednonresidential)145,378addresses  d fromtheirlistingpagesintheU.S.and2,543addressesinPuertoRico.LUCARecanvassfield  J representativescorrected388,838addressesintheU.S.andPuertoRico.  0 AfterparticipatinglocalgovernmentsreceivedfeedbackfromtheCensusBureau,theycould    appealspecificaddresses.Participantsappealed18,442addresses.Approximately54percent   (10,053)oftheaddressesappealedbylocalgovernmentswereincludedonthefinalcensus x  addresslist. ^  TheCensusBureaushouldcontinuetopursueLUCAtypeprogramsinnoncitystyleaddress *z  areasforfuturecensusesandtests.Also,theCensusBureaushouldinvestigatewaystoincrease ` governmentparticipationinLUCAprograms. F  EvaluationF.10,EvaluationoftheUpdate/LeaveOperation (Pennington2003)  InCensus2000,U/Lwasintendedforuseinareaswithsomeaddressesthatwerenotcity-style.  _Noncity_-styleaddresses,suchasRuralRouteandBoxorP.O.Box,areoftennotlinkedtothe t physicallocationofthehousingunit.Whenthereisonlyalocationdescriptionforaunitbutno Z address,maildeliveryofthequestionnaireisnotapossibility.U/Lareaswereprimarilyrural, @ butnottooremoteorsparselypopulated.DesignationofU/Lareaswasmadebyblock.In &v PuertoRico,U/Lwasthesoleenumerationmethod.Questionnaireswithpreprintedaddress  \ labelswerehand-deliveredtoeveryhousingunitontheU/Laddresslist.Existinghousingunits B thatwerenotlistedontheaddressregisteralsorequiredques_tionnaires_,butthesequestionnaires ( werehand-addressedandaddedtotheaddressregister.Sincestaffwereinthefielddelivering  thequestionnaires,theycouldalsomakeotherupdatestotheaddresslistandtothemapsduring  theU/Loperation.ThisevaluationquantifiedtheU/Loperationasonemeansofassessingits   effectivenessandvaluetothecensus-takingprocess. p!  Therewere23,525,257addressesinthestatesideU/Loperationsand1,471,225in <#" PuertoRico.Thesenumbersrepresenttheaddressesthathadeitheralabeledquestionnaireora "$r# handaddressedquestionnaire.Questionnairesweretobedistributedtoallhousingunits %X $ appearinginU/Lareas.SomeoftheaddressesontheU/Laddresslistweredeletedas %>!% nonexistentornonresidentialintheU/Loperationandthelabeledquestionnaireswerenot &$"& delivered.Deletedaddressesareincludedintheworkloadcalculationbecauseittakestimeand ' #' efforttotrytolocatesuchaddresses. (#( StatesideU/Loperationsadded1,644,174addresses,while111,787addresseswereaddedduring l*%* U/LinPuertoRico.Thenumberofcorrectionsinstatesideareaswas9,045,814,with751,156in R+&+ PuertoRico.Somenumberofthesecorrectionsweretotheoccupantnameandtelephone 8,', _numberfields.Thesefieldsquicklybecameoutofdate.Thenumberofdeletes,eitheras  nonexistentorasnonresidential,was1,228,987instatesideareasand122,815inPuertoRico.  SomeunitsthatweredeletedinU/LwerematchedtoU/Laddsafterprocessing,resultingin | 24,265moves,allofwhichwerestateside.UnitsontheaddresslistforU/Lthatdidnotreceive b anyofthesefieldactionsweresaidtobeverified.Therewere11,582,017verifiedaddresses H statesideand485,467verifiedaddressesinPuertoRico. . ~ NoteveryaddressaddedintheU/Loperationwasincludedinthecensus.Somerecordswere  J notincludedbecausetheydidnotcontainsufficientaddressinformationforaddingtothe  0 addresslistordatasufficienttobeassignedtoablock.Otheraddedrecordswerefoundin    subsequentoperationstorepresenthousingunitsthatdidnotexistinthedesignatedblock,either    becausetheunitwasnonexistentorbecausetheunitexistedinanotherblock.Ofthe1,644,174   U/LaddsintheU.S.,85.2percentwereinthefinalCensuscounts.InPuertoRico,83.7percent x  ofthe111,787addedaddresseswereincludedinthecounts. ^  Recommendationsresultingfromthisevaluationinclude: *z  " i"   i[Y23  0    Researchareasthatcouldbeconvertedtoamailout/mailbackmethodology.Theanalysis F revealedlargenumbersofblocksinU/LareasthatwerewhollycoveredontheDSF. i[YY݌,(#(# Ќ  " i"   iZ23  0    Assessthevalueofupdatestotheoccupantnameandtelephonenumberfields. iZZ݌(#(# Ќ  EvaluationF.11,UrbanUpdate/Leave (Rosenthal2002a) t TheCensusBureauconductedtheUrbanU/LoperationfromMarch3toMarch31,2000.The @ objectiveoftheUrbanU/Loperationwastoimprovecoveragebyimprovingthedeliverability &v ofthequestionnairesandupdatingaddressinformationandcensusmaps.TheUrbanU/L  \ operationtargetedareasdeemedunsuitableforMailout/Mailback.Primarily,theseareas B includedmultiunitbuildingswheretheUSPSdeliversthemailtoadroppointinsteadof ( individualunitdesignationsandurbancommunitiesthathadcitystyleaddressesbutmany  residentspickeduptheirmailataP.O.box.TheUrbanU/Loperationreliedonthelocalregions  toidentifyareasbasedontheirknowledgeofwhethertheUSPScouldadequatelydeliverthe   censusquestionnaires.InUrbanU/Lareas,enumeratorsdeliveredthecensusquestionnairesand p!  updatedaddressregistersandcensusmaps,concurrently.Residentswereaskedtocompleteand V"! mailbacktheircensusquestionnaires.EightregionsparticipatedinUrbanU/L:Atlanta,Boston, <#" Chicago,Dallas,Denver,Detroit,Philadelphia,andSeattle.TwelvestatesandDChadUrban "$r# U/Lareas. %X $ Nationwide,12,843blockswerecoveredbyUrbanU/L.Almost60percentoftheseblocks &$"& containedhousingunits.TheMAFhad314,059residentialaddressesinUrbanU/Lblocks. ' #' Afterremovingknownduplicates,therewere310,114addresses.Ofthe310,114addresses, (#( 280,086addresses,or90.3percent,weredeliveredtotheDMAF.Ultimately,238,216 )$) addresses,or85.1percentoftheDMAFaddresses,wereenumeratedinthecensusaseither l*%* occupiedorvacanthousingunits. R+&+ _ 8,', UrbanU/LcontributedtothesuccessofCensus2000byimprovingtheaddresslistand  successfullytargetinghardtoenumerateareas.Ofthe267,005addressesintheaddress  registers,18.1percentwereupdated.Therewere13,131additionsduringquestionnairedelivery, | a4.9percentincreasetotheaddressesprintedintheaddressregisters. b Therewere2,114blockswhere75percentorfewerofthehousingunitsintheblockmatchedthe . ~ DSF.Theseblockscontained15.3percentofthehousingunitsinUrbanU/Lareas.Suchblocks  d wouldpresumablypresentmaildeliverychallengesfortheUSPS.  J Lessthan1percentofUrbanU/Lhousingunitsweredropdelivery;thatis,mailisdeliveredtoa    centrallocationinsteadoftoindividualunitsofamultiunitstructure.Whiletheseaddresses    wereincludedinUrbanU/L,theydidnotmakeupalargepartoftheUrbanU/Lhousingunitsin   thecensus.Furthermore,thevariableusedtoidentifydropdeliverystatusisnotrobust. x  Closetoonequarterofthehousingunitsinthecensuswithhardtocountscoreswereinthe D  hardesthardtocountclass.ThePlanningDatabaseprovideda1990Censustractlevelhardto *z  countscore,acompositemeasureofcharacteristicscorrelatedwithsuccessincountingpeople. ` Evaluatorsclassifiedeachhardtocountscoreintooneoftenhardtocountclasses.Matching F theCensus2000censustractstothePlanningDatabase,189,045addresses,or79.4percentof , theUrbanU/Lhousingunitsinthecensus,wereincensustractsthatcouldbematchedbetween  the1990CensusandCensus2000.  Personsunder18yearsold,AfricanAmericans,andrenterswereoverrepresentedinUrbanU/L t areasascomparedtothenation.Thesetraditionallyundercountedpersonswereenumeratedby Z mailatlowerpercentagesthantheaveragehouseholdorpersonsinUrbanU/Lareas. @ Recommendationsresultingfromthisevaluationinclude:  \ " i"   iKm23  0    DesignateareasforUrbanU/LbasedonheadquartersobjectivessupplementedbyRO ( inputinsteadofthecurrentpracticeoftheregionsdesignatingareasasUrbanU/L. iKmxm݌(#(# Ќ  " i"   in23  0    ConductmorefieldworkorreceivebetterUSPSinputtoidentifydropdeliverystatus. inn݌ (#(# Ќ  " i"   io23  0    ExpanduseofthePlanningDatabasetotargethardtocountareasdeemedsuitablefor V"! UrbanU/L. ioo݌<#"(#(# Ќ  " i "   ip23  0    ConsiderenumeratingtraditionallyundercountedareasusingU/EinsteadofUrbanU/L. ip%q݌%X $(#(# Ќ  EvaluationF.12,Update/Enumerate (Rosenthal2002b) &$"& TheU/Emethodtargetedcommunitieswithspecialenumerationneedsandareaswheremost (#( housingunitsmaynothavehadhousenumberandstreetnamemailingaddresses.Theseareas )$) includedresortareaswithhighconcentrationsofseasonallyvacanthousingunits,selected l*%* AmericanIndianreservations,andcolonias.InU/Eareas,enumeratorsupdatedtheiraddress R+&+ registersandcensusmapsandenumeratedthehousingunitatthetimeoftheirvisit.LCOs, 8,', usinggeneralguidelines,designatedareasforU/E.EveryRegionalCensusCenter(RCC)except  DetroithadareasenumeratedusingtheU/Emethodology.ThirtyfivestateshadU/Eareas.  Nationwide,183,889blockswerecoveredbyU/Eand75,827oftheseblocks(41.2percent) b containedhousingunits.TheMAFhad1,169,090residentialaddressesinU/Eblocks,after H removingknownduplicates.Ofthe1,169,090addresses,90.4percentweredeliveredtothe . ~ DMAF.Ultimately,956,214U/Eaddresses(90.5percentoftheDMAFaddresses)were  d enumeratedinthecensusaseitheroccupiedorvacanthousingunits.  J Thisevaluationlookedattheextentofaddressupdating,descriptivestatisticsoftheaddresses,    demographiccharacteristicsofthehouseholdsandpeoplelivinginU/Eareas,andtimingand    costoftheoperation.   TheU/EoperationcontributedtothesuccessofCensus2000.U/EhelpedtheCensusBureau ^  improvetheaddresslistanddemonstratedthatareassuitedtofieldenumerationwereidentified. D  Ofthe926,861addressesintheaddressregisters,37.2percentwereupdated.Themostfrequent ` updates,corrections(changeintheaddress),weremadeto284,127addresses.Theremainderof F theupdateswerenearlyalldeletions.Therewere129,692U/Eadditionsduringfield , enumeration,a14.0percentincreasetotheaddressesprintedintheaddressregisters.  For71.9percentofblocks,nomorethan25percentofthehousingunitsintheblockmatched  theDSF.Theseblockscontained60.6percentoftheU/Ehousingunits.Suchblockswould t presumablypresentmaildeliverychallenges. Z Oftheaddressesinthecensus,15.2percenthadnoaddressinformation;thatis,thehousingunit &v wasmissingthehousenumber,streetname,ruralroute,andP.O.boxinformation.  \ ThePlanningDatabaseprovideda1990Censustractlevelhardtocountscore,whichisa ( compositemeasureofcharacteristicscorrelatedwithsuccessincountingpeople.Evaluators  classifiedeachhardtocountscoreintooneoftenhardtocountclasses.MatchingtheCensus  2000tractstothePlanningDatabaseB  [programs,4,R1] C  ,59.2percentoftheU/Eaddressesinthecensuswerein   tractsthatcouldbematched.Whileaboutonequarteroftheaddresseswereinthetopthree p!  hardtocountclassesandfewaddresses(0.6percent)wereinthebottomtwohardtocount V"! classes,U/Ewasnotlimitedtothemostdifficulthardtocountclasses.Theseresultsshowthat <#" theCensusBureaufollowedthe1995CensusTestrecommendationtonottargetU/Ebasedon "$r# hardtoenumeratecriteria. %X $ ThehigherthannationalenumerationratesofAmericanIndians/AlaskaNatives,Hispanics,and &$"& vacanthousingunitsB  [AmericanIndians/AlaskaNatives:programs,9B,p.  55,R20,Hispanics:programs,9B,p.55,R21,vacants:programs,15B,p.1,R5;national  rates,U.S.CensusBureau,2001d,R4(AmericanIndians/AlaskaNatives),R5  (Hispanics),R7(vacants)] C  indicatesuccessfultargetingofareaswithspecialenumerationneeds. ' #' TheaveragehouseholdsizeinU/Eareaswas2.9persons,comparedto2.6personsnationally. )$) TheU/Evacancyrateof38.7percentwashigherthanthenationalvacancyrateof9.0percent. l*%* Mostvacantswereseasonalvacants.Ofoccupiedhousingunits,76.1percentwereowned, R+&+ comparedto66.2percentnationally.Ofpersons,49.6percentweremale,comparedto49.1 8,', percentnationally.Ofpersons,31.9percentwereunder18yearsold,comparedto25.7percent  nationally.Ofpersons,23.6percentwereHispanic,comparedto12.5percentnationally.Of  persons,1.5percentwereAfricanAmerican,comparedto12.3percentnationally.Ofpersons, | 27.7percentwereAmericanIndian/AlaskaNative,comparedto0.9percentnationally. b U/EwasconductedfromMarch13toJune5,2000,oneweekpasttheplannedMay30,2000 . ~ enddate.TheextratimewasneededtoenumerateanAmericanIndianreservationatthebottom  d oftheGrandCanyon,accessibleonlybymule.  J  EvaluationF.13,List/Enumerate (Zajac2002)    List/Enumerate(L/E)wasanoperationusedinsparselypopulatedareasofthecountry.Census   enumeratorswereassignedareastocanvassandweregivencensusmapsfortheseareas.The x  enumeratorswereresponsibleforlistingaddresseswithintheirareaonblankaddressregister ^  pages,locatingtheaddressesoncensusmaps(mapspotting),andconductinganinterviewto D  collectcensusinformationforeachaddress.Theoperation,whichincludedreinterviewandfield *z  followupcomponents,wascarriedoutfrommidMarch2000tothebeginningofJuly2000. ` ThisevaluationexaminedthecharacteristicsofaddressesaddedtotheMAFbyL/E. F L/Ewasresponsibleforadding392,368addressesnationwidetotheMAF.Oftheseaddresses,  391,276mettheeligibilitycriteriatobeinthecensus.Thisisabout99.7percentofalladded  L/Eaddresses.Oftheaddresseseligibletobeinthecensus,389,749addresseswereactually  includedinthefinalcensuscount.Thisrepresents99.6percentoftheeligibleL/Eaddressesand t 99.3percentofalladdedL/Eaddresses. Z Atotalof47,927blockshadatleastoneL/Eaddress.Oftheseblocks,4.7percenthadallof &v theiraddressesrecognizedbytheUSPS.Thisindicatesthattheseblockscouldhavepossibly  \ beenconvertedtothe_Mailout_/Mailbackenumerationmethodology.Theseblockscontained1.4 B percentoftheaddressesaddedduringL/E. ( L/Eappearstobesuccessfulforthefollowingreasons:   * #$%&'((xx * #$%&'(" i"   i׏23  0    Coverage:Atotalof392,368addresseswereaddedfromtheoperation. i׏݌p! (#(# Ќ  " i"   iԐ23  0    FutureLocatabilityofAddresses:Over50percent(197,525ofthe392,368)were <#" completecitystyletypeaddresses.Ofthe160,232addressesthatwerenotcomplete "$r# citystyleornotcompleteruralroute,85.2percenthadlocationdescriptioninformation. %X $ Inaddition,98.7percentofalladdedL/Eaddresseshadavalidmapspot. iԐ݌%>!%(#(# Ќ  " i"   i23  0    QualityofAddresses:About99.3percentofthe392,368addressesmadeitintothe ' #' census. i?݌(#((#(# Ќ  " i"   i823  0    TargetingofAreas:Only1.0percentoftheblocksinL/Ehadalloftheiraddresses l*%* recognizedbytheUSPS.Theseblocksrepresentjust1.4percentoftheaddressesadded R+&+ duringtheoperation. i8e݌8,',(#(# Ќ  EvaluationF.15,AnAssessmentofAddressesontheMAF"Missing"intheCensusor  GeocodedtotheWrongCollectionBlock (Ruhnke2003)  OneoftheresultsoftheAccuracyandCoverageEvaluation(A.C.E.)includedarepresentative b sampleofaddressesthatwerecodedas"missing"fromthecensus.Thisevaluationconducted H additionalresearchtobetterunderstandthese"missing"addressesandtoexaminethereasonsfor . ~ theirstatusof"missing"aftertheA.C.E.FinalHousingUnitworkwascompleted.Evaluators  d matchedtheaddressescodedas"missing"toallnonduplicatehousingunitsontheMAFina  J largergeographicsearchareathantheoneusedbytheA.C.E.andsearchedformatchesinthe  0 tractwhichincludedeachaddressandinallsurroundingtracts.Themainfocusinunderstanding    these"missing"addresseswastodetermineiftheywereactuallyincludedinthecensusas    housingunits,butwereincorrectlygeocodedtoacollectionblockoutsideoftheA.C.E.   geographicsearcharea.Sincematchingwasnotlimitedtocensusaddresses,butincludedall x  nonduplicatehousingunitsontheMAF,evaluatorswereabletoexamineaddressesthatwereon ^  theMAFortheDMAFbutwereexcludedfromthecensus. D  About8,900ofthesampleunitscodedas"missing"bytheA.C.E.werematchedtounitsonthe ` MAFduringthisevaluation.About4,800ofthemwerematchedtoaddressesthatwereincluded F inCensus2000.Ofthosecensusmatches,about3,100weregeocodedinerrorinthecensustoa , collectionblockthatwasdifferentthantheblockprovidedbytheA.C.E.Theother1,700units  werematchedtocensusaddressesthatweregeocodedtothesameblockastheA.C.E. missing  addresses.Therearetwoprimaryreasonsthatthesecensusunitswerenotincludedinthecensus  addresslistusedfortheA.C.E.addressmatching. t Thefirstreasonisthatsomeoftheseunitswereidentifiedaspotentialduplicatesduringthe @ Census2000HousingUnitUnduplicationoperationsandwerethereforekeptoutoftheA.C.E. &v FinalHousingUnitmatchingoperation.About78percentofthematchestoincensusunitsin  \ thesameblockwerepotentialduplicatesthatultimatelywerereinstatedinthecensus.The B remaining22percentoftheincensusmatchestoA.C.E. missingunitsinthesameblockwere ( notreinstatedduplicates.Areasonthattheseunitswereexcludedfromtheaddresslistusedfor  theA.C.E.addressmatchingisthattheywerenotgeocodedtoanA.C.E.sampleblockatthe  timeoftheFinalHousingUnitmatching,butweremovedintoanA.C.E.sampleblockintime   forevaluationwork. p!  Oftheapproximately8,900sampleaddressescodedas"missing"bytheA.C.E.thatmatchedto <#" theMAFinthisevaluation,about4,000werenotincludedinCensus2000.Thatis,theseunits "$r# werelistedandconfirmedasgood,residentialaddressesduringtheA.C.E.,buttheCensus %X $ BureausrulesforcreatingtheDMAFandtheHundredPercentCensusUneditedFileexcluded %>!% themfromthecensus.Thoseunitsrepresentaweightedestimateof1.3millionunitscodedas &$"& erroneouslyexcludedfromthecensusasmeasuredbytheA.C.E.andthisevaluation. ' #' About28percentofthecasescodedaserroneouslyexcludedunitswereneverdeliveredtothe )$) DMAF.ThereareanumberofreasonsunitsontheMAFwouldhavenotbeensenttothe l*%* DMAFasaresultoftheCensusBureausrulesfordevelopingtheCensus2000addressframe. R+&+ OneofthereasonsaunitwouldnotbeincludedontheDMAFisifitwascodedbytheUSPSas 8,', nonresidentialontheDSF.TheCensusBureauexcludedthoseaddressesfromtheoriginal  censusaddresslistbecauseitwouldnotbeprudenttomailquestionnairestoallnonresidential  addresses.TheCensusBureaureliedonfieldlistingoperationstoaddthoseunitsiftheywere | actuallyresidentialunitsbyCensusBureaudefinitions. b About49percentofthecasescodedaserroneouslyexcludedunitswereontheDMAF,butwere . ~ deletedduringtheKillProcess.ThegoaloftheKillProcesswastoidentifyunitsthatweremost  d likelybadaddresses(forexample:aunitforwhichnocensusformwasreceivedandtheunitwas  J deletedinboththeNRFUandCIFUoperations)andremovethemfromthecensus.  0 About22percentofthecasescodedaserroneouslydeletedunitswereontheDMAFbutwere    determinedtobepotentialduplicatesduringtheHousingUnitUnduplicationoperationsthrough   addressandpersonmatchingalgorithms.TheCensusBureauultimatelydecidedtoexclude x  thoseunitsfromCensus2000.TheamountoferroneousdeletionsfromtheUnduplication ^  operationasmeasuredinthisevaluationispotentiallyoverstated.Thiscomesfromthefactthat D  theA.C.E.mayhavecodedsomethingasmissingfromthecensus,whenitwasactuallyincluded *z  inthecensuswithadifferentformoftheaddress.TheUnduplicationoperationmayhave ` recognizedtheduplicationbutremovedtheversionoftheaddressthattheA.C.E.listed. F TheestimatedpercentageofcensusaddressesthatweregeocodedtotheincorrectCensus2000  collectionblockis4.8percent(standarderroris0.3percent).Theestimatedpercentageof  geocodingerrorinthecensuswassignificantlyhigherin_Mailout_/Mailbackenumerationareas  (5.5percent)thaninU/L(1.7percent)orL/Eareas(1.2percent). t Geocodingerrorwasmoreprevalentamonghousingunitsinmultiunitstructures.Housing @ unitsinbothsmallandlargemultiunitstructureshadasignificantlyhighergeocodingerror &v estimatethansingleunitsorhousingunitsintwounitstructures.Additionally,largemultiunits  \ (housingunitsinstructureswithtenormoreunits)hadasignificantlyhighergeocodingerror B estimatethansmallmultiunits(housingunitsinstructureswiththreetonineunits).The ( geocodingerrorestimateforbothsinglehousingunitsandtwounitstructureswasabout3  percent,forsmallmultiunitstructureswasabout5percent,andforlargemultiunitstructures  wasabout11percent.Geocodingerrorsareexpectedtobehigherforunitsinmultiunit   structuresbecausegeocodingerrorisastructurebasedproblem.Geocodingthestructuretothe p!  wrongblockcauses_every_Ԁunitinthatstructuretobegeocodedtothewrongblock.Thelarger V"! thestructure,thelargerthenumberofgeocodingerrorcasestherewillbeifthestructureis <#" geocodedtotheincorrectblock. "$r# Geocodingerrorofcensusaddresseswaslessfrequentincertainregionsofthecountry.The %>!% geocodingerrorestimatefortheMidwest(3.8percent)wassignificantlylowerthanthe &$"& geocodingerrorestimatefortheSouth(5.7percent).Therewerenoothersignificant ' #' differences.GeocodingerrorestimatesalsodifferedforsomeoftheROs.TheBostonand (#( KansasCityROsbothhadasignificantlylowergeocodingerrorestimatethanthenational )$) estimateof4.8percent. l*%*  8,', Oneofthereasonsaddresseswerecodedas"missing"fromthecensusduringtheA.C.E.was  becausetheywereincorrectlygeocodedinthecensustoacollectionblockoutsideofthescope  ofthe_A.C.E._'sgeographicsearcharea. | Recommendationsinclude: H " i"   i͸23  0    ResearchthepossibilityofcollectingGlobalPositioningSystemcoordinatesfor  d addressesinthecensus,tohelpenumeratorsfindtheirassignmentsandtoensure  J geocodingunitstothecorrectblock.Considergettingbettergeocodingfortheareaswith  0 thehighestgeocodingerrorrates,whichare_Mailout_/Mailbackareas.Thisresearchis    currentlyunderway. i͸݌  (#(# Ќ  " i"   i]23  0    Conductresearchtorefineproceduresforidentifyinganddeletingunitsbelievedtobe x  duplicates.Theunduplicationprocessappearstohavedeletedmanyunitswhichshould ^  havebeenincludedinthecensus.Workhasalreadybegunonbuildinganunduplication D  processintothe2010Census. i]݌*z (#(# Ќ  EvaluationF.16,EvaluationoftheBlockSplittingOperationforTabulationPurposes  F (GreenandRothhaas2004) , Thisevaluationmeasuredthepercentofthecountryaffectedbycollectionblockssplitfor  tabulationpurposesandtheaccuracyofthatblocksplitting.Collectionblocksaregeographic  areasthatareusuallydefinedbyvisiblefeatures,andusedbytheCensusBureautoconductfield t operations.Often,collectionblockscrossgovernmentalunitboundariessuchascityandtown Z orotherrequireddatatabulationboundaries.AttheendofCensus2000,theCensusBureau @ redefinedthecensuscollectionblocksbyrecognizingtheboundariesofgovernmentalunitsand &v othergeographicentitiesrequiredfortabulationofcensusdata.Oneofthestepsneededto  \ achievethisinvolvedusinganautomatedsystemtosplitcollectionblocksincertainsituations. B Thisblocksplittingprocesswasbasedonaddressrangesandmapspotinformationinthe ( TIGERdatabase.Toevaluatetheblocksplittingprocess,evaluatorsselectedasampleof1,000  collectionblocksthathadatleastonetabulationboundarythatsplittheblockforfield  verification.Fieldrepresentativesdeterminedwhetherthehousingunitsintheseblockswere   allocatedtothecorrectsideorthewrongsideofeachtabulationboundary. p!  About916,000blocksoutofthe5.1millionblocksinthecountryweresplitfortabulation <#" purposes.Atotalof282,457blocksformedthesamplinguniverseusedtoevaluatetheblock "$r# splittingprocess.The633,337splitblocksexcludedfromthesamplinguniversewereeither %X $ locatedinremoteAlaska,locatedinPuertoRico,weresplitbytheboundariesofspecialpurpose %>!% governmentaloradministrativeentitiessuchasschooldistricts,weresplitbytheboundariesof &$"& statisticalentities,orcontainednohousingunitsorgroupquarters.RemoteAlaskaandPuerto ' #' Ricowereexcludedfromtheevaluationtominimizecost.Boundariesofspecialpurpose (#( governmental,administrative,andstatisticalboundarieswereexcludedbecausethisevaluation )$) reliedontheknowledgeofresidentsoftheblockandtheywouldnotnecessarilyknowwhere l*%* thesetypesofboundariesexistedintheirblocks.Splitblocksthatcontainednohousingunitsor R+&+ groupquarterswereexcludedbecausethepurposeoftheevaluationwastomeasuretheerror 8,', associatedwithplacinghousingunitsandgroupquartersinthewrongtabulationblock.The  estimatednumberofblocksthatfellintoeachofthesecategorieswasnotavailable,butthesum  totalwas633,337blocks.Alittlemorethan10percentofthe115.5millionhousingunitsinthe | countrywerelocatedinthesplitcollectionblocksinthesamplinguniverse. b Resultsshowedthatover26percentofthesesplitcollectionblocksinthesamplinguniverse . ~ containedatleastonehousingunitallocatedtothewrongsideofthetabulationboundary.  d Althoughthispercentageishigh,splitcollectionblockswithatleastonehousingunitallocated  J tothewrongsideofatabulationboundaryrepresentlessthan2percentofthecollectionblocks  0 inthecountry.Forhousingunits,about3.65percentofthe12millionhousingunitsinthesplit    collectionblocksinthesamplinguniversewereallocatedtothewrongsideofatabulation    boundary.Theseerrorsrepresent0.37percentofthehousingunitsinthecountry.Forthegroup   quartersinthesample,nonewereallocatedtothewrongsideofatabulationboundary. x  Althoughtheestimateofthenumberofgroupquartersallocatedtothewrongsideofatabulation ^  boundarywaszero,theCensusBureaucannotconcludethattherewerenogroupquartersin D  errorthroughoutthecountry.ThepreliminaryAugust2002resultsfromanadministrative *z  program,inwhichtheCensusBureaureceivesinputfromlocalgovernmentalentities,showed ` that1,867groupquartersinthecountrywereinfactallocatedtothewrongsideofatabulation F boundary.Thisislessthan1percentofallgroupquarters. , Forthisevaluation,collectionblockswerecategorizedinthe_mailout_/mailback,military,Urban  U/L,andUrbanU/Eenumerationareasas insidetheblueline.Theterm insidetheblueline  referstoareaswherealmostallmaildeliverywastocitystyleaddresses.ExceptforRemote t Alaska,allothertypesofenumerationareaswerecategorizedas outsidetheblueline.This Z termreferstoareaswheremaildeliverywasto_noncity_Ԅstyleaddresses.Amixtureofcitystyle @ and_noncity_Ԅstyleaddressesoccurinsometypesofenumerationareas,especiallythose outside &v theblueline.Forthehousingunitsaffectedbyblocksplitting,thepercentinerrorfor  \ enumerationareas insidethebluelinewascomparabletothepercentinerrorforenumeration B areas outsidetheblueline ( Forhousingunitsaffectedbyblocksplitting,fewerthan4percentofthe11.1millionhousing  unitswithcitystyleaddressesandfewerthan3.5percentofthe773,000housingunitswith   _noncity_Ԅstyleaddresseswereallocatedtothewrongsideofatabulationboundary.Thepercent p!  inerrorforbothaddresstypesinthecountrywasthesame(0.37percent). V"!   %X $ _ CategoryGFieldRecruitingandManagement   SummariesofthetwopartsofEvaluationG.1intheFieldRecruitingandManagementcategory | follow. b  EvaluationG.1,Part1,Census2000StaffingPrograms,RecruitingComponent (Jacobson . ~ andPetta2002)  d Thisevaluationdescribedthefactorsthataffectedrecruitingperformancein519ofthe520  0 LCOsduringCensus2000.l '  1      ׀Theeffectofseveraldifferenttypesoffactorswereexamined:    censuspayrelativetolocallyprevailingpay;recruitinggoalssetbyheadquarters,area    characteristicssuchaspopulationdensity,privatefirmemploymentandpercapitaincome,and   managersstartdateandturnover.ThisworkwasmodeledonWestatssimilaranalysisof x  enumerationperformanceduringthe1990decennialcensus. ^  TheanalysisoffactorsaffectingCensus2000recruitingproducedseveralimportantconclusions *z  thatwerestronglysupportedbyregressionanalysis,inkeepingwithreasonableexpectations, ` andconsistentwiththeobservationsofknowledgeableobservers.Themostbasicfindingswere F thattherewasconsiderablevariationinrecruitingperformanceacrossLCOs.Factorsthat , accountedforabouthalfofthatvariationfallintothreecategories:pay,areacharacteristics(such  astheexpectednonresponseworkloadandnumberofworkersavailabletoberecruited),and  managementfactors(suchasturnoverofLCOmanagers).  Enumeratorpay,relativetolocallyprevailingpay,wasakeydeterminantofrecruiting Z performance.Theoverallhighlevelsofpaycomparedto1990greatlyfacilitatedrecruiting,but @ itwasnotpossibletoeliminateallcrossLCOvariationinrelativepay.Relativepaywasabout &v 77percentofprevailingpayinLCOswithmuchbelowaveragerecruitingperformanceand  \ about84percentofprevailingpayinLCOswithmuchaboveaverageperformance.That B variationwasassociatedwitha10percentincreaseinthenumberofapplicantsintheLCOswith ( highversuslowrelativepay.Thisisanimportantfindingsinceitsuggeststhatpayincreases  couldhavebeenreliedupontofurtherenhancerecruitingperformancehadthatbeennecessary.  Intermsofareacharacteristics,theNRFUworkloadstronglyinfluencedrecruiting.Onaverage, p!  theNRFUworkloadwasabout80,000casesperLCO,butmanyLCOshadfewerthan56,000 V"! casesandmanyhadmorethan104,000cases.Anincreaseincaseloadof24,000,aboutone <#" standarddeviation,wasassociatedwitha13percentincreaseinqualifiedapplicants.Thisresult "$r# suggeststhatLCOsappropriatelyattemptedtosecureenoughapplicantstofilltherequired %X $ numberofenumeratorslots,butotherfactorsverystronglyaffectedrecruitingsuccess. %>!% ThenumberofworkersemployedbyallfirmswithinaLCOareahadapositiveeffecton ' #' recruiting,buttheeffectwassmallgiventhatemploymentlevelsinLCOsshowedhuge (#( variation.ThissuggeststhattherewereplentyofindividualsinvirtuallyeveryLCOsareawho  couldbeattractedtoapplyforcensusjobsanditwasnotnecessarytotakevariationinthe  numberofpeoplewhocanbedrawnuponinallocatingrecruitingresources. | Anunanticipatedresultwasthataonestandarddeviationincreaseintestscores,anincrease H from85.6to88.4,wasassociatedwithadecreaseinthenumberofapplicantsofalmost11 . ~ percent.Thisresultmayhavebeenduetorecruitingbeingmoredifficultinareaswheremany  d peoplewerelikelytodowellonthetest(evenholdingrelativewagesandpercapitaincome  J constant).Theeffectwassufficientlylargethatvariationintestscoresshouldbetakeninto  0 accountinsettingwagesandmanagersshouldanticipatethatareaswithhightestscoreswill    needadditionalassistanceinmeetingrecruitinggoals.    Intermsofmanagementfactors,resignations,terminationsforcause,orleavingforanyother x  reasonbyLCOmanagementduringtherecruitingperiodwereassociatedwithareductioninthe ^  numberofrecruitsbyabout12percent.ThisevidencestronglysuggeststhatLCOmanagement D  performanceisakeydeterminantofrecruitingsuccessandthatLCOmanagementneedstobein *z  placeforasubstantialperiodtobehighlyeffective. ` Byfar,thelargestsourceofvariationinrecruitingperformancewasassociatedwithaLCO , beinginoneofthreeCensus2000administrativeregions.Evenaftertakingotherkeyfactors  intoaccount,LCOsintheSeattleandDenverregionsrecruitedaboutonethirdmoreapplicants  andLCOsintheNewYorkregionrecruitedabout17percentmoreapplicantsthanLCOsin  otherregions. t  EvaluationG.1,Part2,Census2000StaffingPrograms,PayComponent (Jacobsonetal @ 2002) &v Thisevaluationdescribedanalysisoftheeffectofpay,frontloading,andotherfactorsonhow B quicklytheCensus2000NRFUwascompleted.FromlateApriltolateJuneof2000,thehouses ( ofpeoplewhofailedtoreturntheircensusformswerevisitedbyabout510,000enumerators.  TheevaluationexamineddifferencesinhowquicklytheNRFUwascarriedoutintheLCOs;  differencesinhowquicklytheNRFUwascompletedrelativetothe1990CensusNRFU;andthe   underlyingreasonsforthosedifferences.Overall,theCensusBureausplantoraisewagestoat p!  least75percentoflocallevelsandtoputtoworkduringthefirstweektwicethenumberof V"! enumeratorsthatwouldbeneedediftherewerenoattrition,directlyledtodramatic <#" improvementinspeedrelativetothe1990NRFU. "$r# Hourlypaywasincreasedby37.8percentonaveragerelativeto1990(adjustedforinflation)and %>!% theassociatedincreaseinenumeratorretentionwas22.6percent.Thisincreaseinretention, &$"& coupledwithintroducingfrontloading(increasingthenumberofenumeratorsatworkatthe ' #' outsetrelativetocasestocomplete),permittedtheaverageCensus2000LCOtocompletethe (#( NRFUin7.19weekscomparedto9.72weeksin1990.Moreover,theslowestperformingLCOs )$) completedtheirworkabout1.5weeksfasterthanthefastestperformingLCOin1990. l*%* _ 8,', Analysisofthevariationincompletiontimeacrossthe510LCOswithadequatedata(outofa  totalof520LCOs)showsthatdifferencesinthedegreeoffrontloadingwastheprimarysource  ofvariationincompletiontime,differencesinthenumberofcasescompletedbyindividual | enumeratorsplayedonlyasmallrole,anddifferencesinretentionofindividualenumerators b weretoosmalltohavemuchofaneffect. H AnalysisoftheinfluenceoffactorswithinandoutsideofthecontroloftheCensusBureau,using  d administrativedatabasescoveringthe510LCOsplusasurveydatabasecoveringcloseto2,800  J enumeratorsin376crewsin27LCOs,showedthattheNRFUwascompletedmostrapidlyin  0 lowwageareasandareaswhereapplicantstestscoreswerelowonaverage,intheDenverand    LosAngelesCensusRegionswheremanagersensuredthathighlevelsoffrontloadingwere    achieved,andinLCOsthathadfewercasestocomplete(relativetolargerscaleoffices)andin   whichmanagersdidnotturnover. x  Thefirstoverallconclusionfromthisanalysiswasthatdifferencesinfactorsoutsideofthe D  controlofcensusmanagers,suchasthelaborforceandareacharacteristics,hadsmalleffectson *z  completiontimeandproductivity.Incontrast,factorslargelywithincensusmanagement ` control,suchasthetotalnumberofLCOs,thenumberofcasestocompletewithinagivenLCO, F censuspaylevels,andRegionalOffice(RO)planningandoversight,hadlargeeffectson , performance.Wherethebasicpay,recruiting,andfrontloadingplanswerefollowed,LCOs  succeededinsecuringandretainingmorethanenoughapplicantstostafftheNRFUwithhighly  competentenumeratorswhoalsowerestronglymotivatedtoworkaslongasneeded.  Thesecond,butsinglemostimportant,conclusionwasthatthedegreetowhichLCOsexceeded Z scheduleswaslargelyafunctionoftheamountoffrontloadingtheyachieved.About80percent @ oftheLCOsmetorexceededfrontloadinggoals.However,theroughly20percentoftheLCOs &v thatdidnotmeettheirfrontloadinggoalstookabouttwoadditionalweekstoreachtheirweek1  \ goals.Understandingwhyfrontloadinggoalswerenotmet,therefore,isthekeyto B understandingthesourceofvariationinspeed. ( Oneormoreofthefollowingthreehiringexplanationsprobablyledtorecruitingshortfalls:  hiringwasinherentlymoredifficultduetofactorsoutsideofcensusmanagementcontrol,hiring   wasnoteffectivelymanaged,andmanagersdidnotfeelitwasessentialtomeetfrontloading p!  goals.Unfortunately,thedataneededtodefinitivelysortouttherelativeimportanceofthethree V"! explanationswerelacking <#" Thethird,butsomewhatspeculative,conclusionwasthatimprovementsinthehiringprocess %X $ wereneededtomeetfrontloadinggoals.Possibleimprovementsincludestartingthehiring %>!% processearlierandensuringenoughhiringclerksandphonelinesareavailabletooffset &$"& unexpectedhiringdifficulties.Thus,muchasfrontloadingofenumeratorswasthekeyto ' #' dramaticallyincreasingthespeedinconductingtheenumeration,increasinghiringcapacity (#( appearstobethekeytomeetingfrontloadinggoals. )$) Afourthconclusionwasthatsettingpaycompetitivelywasessentialtorecruitingsufficient R+&+ numbersofwellqualified_applicants_Ԁandtoretainingenumeratorsaslongastheywereneeded. 8,', _ԇTheseresultssuggestthatcensuspayexceededthethresholdabovewhichpeoplewhoagreedto  acceptenumeratorpositionsweresufficientlycompetenttoexecutetheworkandwouldnot  lightlybreaktheircommitmentstoworkwhiletheirserviceswererequired.About90percentof | theCensus2000enumeratorsshowedthemselvestobehighlyproductive,asmeasuredbythe b numberofcasestheywereabletocompleteperhour.Incontrast,duringthe1990Census H NRFU,50percentormoreoftheenumeratorshaddifficultycompletingassignmentsand/orquit . ~ beforecompletingeventheirinitialassignment.  d Thesharpcontrastbetweenpayandperformancein2000versus1990hasseveralimportant  0 implications.PerhapsthemostimportantisthattheCensusBureaushouldreassesshowtest    scoresandavailabilitytoworkmanyhoursareusedashiringscreens.Theanalysissuggeststhat    thecapacitytoquicklycompletetheNRFUwouldhavebeenenhancedhadtestscoresofabout   82percentbeenusedasathreshold(unlessapplicantshadsomespecialskillsuchasfluencyina x  foreignlanguage)andthecontactorderbeenbasedonhoursofavailability(reportedin ^  applications). D  Theequationsproducedinthisevaluationcouldbeextendedtosetthescheduleandthedegree ` offrontloadingforthe2010CensusNRFUinawaythatwouldsubstantiallyreducecostwithout F reducingtheprobabilitythescheduleismet. ,  CategoryHFieldOperations   SummariesofthesevenevaluationsintheFieldOperationscategoryfollow. Z  EvaluationH.2,AssessmentofFieldVerification (Tenebaum2001) &v ThisoperationalassessmentfocusedontheBeCounted/TQAFieldVerificationoperation. B Enumeratorsvisitedthelocationofunitswithoutaconfirmedcensusaddresstoverifytheir ( existencebeforeCensus2000includedtheaddresses.Theoperationalsoincludedaddresses  deletedintwoormorepreviousoperationsbutforwhichtheCensusBureaureceivedamail  return(doubledeleteswithamailreturn).Iftheenumeratorlocatedtheaddress,he/sheentered   acheckmarkontheassignmentlistingforeachunitverifiedasaresidentialaddress.Ifthe p!  addresswasnotalivingquartersorwasaduplicateofanotheraddressontheassignmentlisting, V"! theenumeratorcodeditaccordingly. <#" ThenonIDquestionnaireprocessforCensus2000wasaverycomplexoperationconsistingof %X $ manycomponents.Thisoperationalassessmentonlydiscussedoneaspectofthatoverall %>!% process,thatis,theverificationofaddresseswhichcouldnotbematchedtotheMAFbutcould &$"& begeocodedtoacensusblock.Noconclusionscanbemaderegardinganyothercomponentof ' #' thenonIDquestionnaireprocess. (#( Enumeratorsprovidedinformationaboutthe884,896assignedaddresses.Theycoded51 l*%* percentoftheassignedaddressesasvalidlivingquarters,35percentasnonexistent,and14 R+&+ percentas_duplicates. 8,', ЇOverall,49.18percentoftheaddresseswithoutaconfirmedcensusaddress(thenonIDcases)  werecodedasvalidcensusaddresses.Overall,52.86percentoftheaddressesdeletedintwoor  morepreviousoperationsbutforwhichtheCensusBureaureceivedamailreturn(thedouble | deletes)werecodedasvalidaddresses. b TheCensus2000proceduresprovidedagoodmodelforplanningafieldverificationoperation . ~ forCensus2010,withthefollowingrecommendations:  d " i"   i23  0    RedesigntheFieldVerificationprocedurestocaptureenoughinformationforduplicates  0 toprovidealinkbetweenthe_two_Ԁaddresses. i݌  (#(# Ќ  " i"   i 23  0    Clarifytheproceduresconcerninghowfartoextendthesearchforassignedaddressesso   enumeratorsdonoterroneouslydeleteaddresseslocatedinadjacentblocks. i 6݌x (#(# Ќ  " i"   ix23  0    Conductfurtherresearchintothesourcesofthedoubledeleteswithmailreturnssince D  abouthalfofthemwerecodedasvalidaddresses. ix݌*z (#(# Ќ  " i "   i23  0    ConsiderawaytoindependentlyvalidatetheaccuracyoftheresultstodetermineifField F Verificationimprovesthecensusfiles. i݌,(#(# Ќ  " i "   i23  0    Determinetheeffectthatadditionalresponseoptionsin2010mighthaveonField  Verification. iF݌(#(# Ќ  EvaluationH.4,QuestionnaireAssistanceCentersforCensus2000 (JonesandBarrett2003) Z QuestionnaireAssistanceCentersweretargetedlocationsdesignedtoassistindividualswhohad &v questionsaboutcompletingtheirCensusquestionnaires,whoneededlanguageassistanceon  \ theirquestionnaires,whohadageneralquestionaboutthecensus,orwhoneverreceiveda B censusquestionnaire.ThesecenterswereopenbetweenMarch8,2000andApril14,2000. ( CensusBureauPartnershipSpecialists,inconsultationwithlocalofficialsplayedanimportant  roleinselectingthecensustractswhereQuestionnaireAssistanceCenterswereplaced.Mostof   thetractschosentohaveQuestionnaireAssistanceCenterswereinareasknowntobeeither p!  difficulttoenumerate,heavilypopulatedbycertainracialandethnicgroups,orinlinguistically V"! isolatedareasknowntobeheavilypopulatedbyspeakersofcertainforeignlanguages.Publicly <#" accessiblelocationssuchascommunitycentersandsocialservicecentersweresetuptohouse "$r# QuestionnaireAssistanceCenters.Thecenterswerestaffedbypaidclerksandvolunteers. %X $ Someofthepaidclerkshadforeignlanguageskills,sotheycouldprovideexpertassistanceto %>!% potentialcensusrespondentsexperiencinglanguagedifficulties.Volunteerswerechosenfrom &$"& localcommunitygroupsorotherorganizationsthatwereinpartnershipwiththeCensusBureau. ' #' Bothpaidandunpaidstaffprovidedliteracyassistancetothoserespondentsinneedofit.Staff (#( wereinstructedtocompleteaRecordofContactforeachpotentialcensusrespondentthatvisited )$) thecenter.TheRecordofContactdocumentedthetypeandextentofassistanceneeded. l*%*  8,', _Therewere23,556QuestionnaireAssistanceCentersestablishedduringCensus2000.However,  datawerecollectedandprocessedfromonly14,222.Therewerenodataprocessedfromthe  remainingcenters.Datawerekeyedfor559,027potentialcensusrespondentsthatutilizedthe | QuestionnaireAssistanceCentersduringCensus2000.About39.4percentoftheserespondents b wereprovidedwithaBeCountedForm.Somerespondents(26.4percent)neededassistanceon H aspecifictypeofquestionnaire.Ofthosewhodidneedassistance,mostaskedforhelpin . ~ completingtheshortform.Censusformswereprintedinsixdifferentlanguages:English,  d Spanish,Chinese,Korean,Vietnamese,andTagalog.Ofthepeoplewhoneededassistanceona  J specificquestionnaire,most(64.6percent)requiredassistanceontheEnglishshortform.  0 Respondentsgenerallydidnotaskforassistanceonspecificquestionsorspecificcensusforms.    Onlyafewrespondents(2.3percent)askedforassistancewithapopulationorhousingquestion   ontheircensusform.Only5.8percentofrespondentsindicatedneedingassistancewith x  language,butanother4.6percentmarkedthattheycouldnotreadorunderstandtheform. ^  Respondentsreportedotherquestionnairerelatedproblems,suchasrespondentdidnotreceivea D  censusform,respondentlosttheirmailedcensusform,orrespondenthadaproblemwiththeir *z  questionnairethatwasnotlistedontheRecordofContact. ` TherewerenumerouswaysthatpeoplelearnedabouttheQuestionnaireAssistanceCenters. , About32.6percentoftherespondentslearnedofthecentersinwaysunlistedontheRecordof  Contact.Ofthealternativeslistedontheform,15.5percentreportedhavingseenaposter  announcement.  LanguageAssistanceGuideswereavailablein37differentlanguages.Mostrespondents(83.1 Z percent)didnotrequestaLanguageAssistanceGuide.Ofthe94,639peopleneedingLanguage @ AssistanceGuides,morethanhalf(53.0percent)neededtheminSpanish.BeCountedForms &v wereavailableinsixdifferentlanguages,includingEnglish.Mostrespondents(60.6percent)  \ didnotrequestaBeCountedForm.Ofthosewhodid(220,489people),mostrequestedthemin B English(69.8percent)orSpanish(24.4percent). ( TheCensusBureaushouldcontinuetoestablishQuestionnaireAssistanceCentersduringa  censustohelprespondents.However,thefollowingchangesarerecommended:   " i "   i3023  0    IncreasethenumberoflanguagesinwhichtheCensusBureauprovidestheBeCounted V"! Form.Morethan1,000respondentsrequestedLanguageAssistanceGuidesinlanguages <#" forwhichaBeCountedFormwasnotavailable. i30`0݌"$r#(#(# Ќ  " i "   i123  0    CollectdemographicinformationaboutthecensusrespondentsthatutilizeQuestionnaire %>!% AssistanceCenters.Thiswouldmakeiteasiertotellifthecentersservedthetargeted &$"& population. i12݌' #'(#(# Ќ   )$)  EvaluationH.5,NonresponseFollowupforCensus2000 (Moul2002)  TheobjectiveofNRFUwastoobtaincompletedquestionnairesfromhouseholdsinmailback | areasthatdidnotrespondbymail,throughtheInternet,ortoaTQAoperator.Ifaquestionnaire b wasnotcheckedinbeforetheNRFUuniverseselectionprocessbegan,thehousingunitwas H targetedforNRFU.Therewere119,090,016housingunitsinmailbackareas(includingPuerto . ~ Rico)thatwerepotentiallyeligibleforfollowup.Theinitialworkloadof44,928,883housing  d unitswasidentifiedonaflowbasis.ALateMailReturnoperationsubsequentlyidentified  J 2,555,918housingunitsthatwerecheckedinaftertheinitialuniversewasidentified.The  0 resultingworkload,whichincludedPuertoRico,was42,372,965(35.6percentoftheeligible    universe).TheNRFUoperationwasscheduledtooccurfromApril27throughJuly7,2000.    ActualstartandfinishdateswereApril27,2000andJune26,2000,respectively.   Basedonthefollowing,NRFUwasasuccess. ^  " i "  -X, XX!- i823  0    NRFUofficiallyendedearlyonJune26,2000,tendaysaheadofschedule. i88݌*z (#(# Ќ  " i"   i923  0    Lessthan0.1percentoftheworkloadhadanundeterminedstatusattheendofB  C  NRFU. i99݌F(#(# Ќ  " i"   i:23  0    Comparedtothe5.0percenttarget,therewasalowfinalattemptrate(approximately2.7  percent). i:;݌(#(# Ќ  However,theNRFUoperationwasnotperfect.Forexample: t " i"   ic<23  0    For5.4percentofthereturns,enumeratorsfailedtoindicatewhethertheinterviewwaswith @ ahouseholdmemberoraproxy. ic<<݌&v(#(# Ќ  " i"   i=23  0    Ofthe26.4millionoccupiedhousingunits,0.4percenthadnopopulationcount. i==݌B(#(# Ќ  " i"   i>23  0    Approximately4.2millionhousingunitswereenumeratedmultipletimes.Approximately  3.5millionofthesewereenumeratedinNRFUandbyapapermailreturnquestionnaire. i>>݌(#(# Ќ  ݀   " i"   i@@23  0    Somehousingunitshadanunrealisticallylargenumberofcontinuationformsattached,as p!  manyas99. i@@m@݌V"!(#(# Ќ  Ofthe42.4millionhousingunits,62.3percentwereoccupied,23.3percentwerevacant,and "$r# 14.3percentweredeleted.Approximately79.6percentoftheoccupiedreturnswerecompleted %X $ byahouseholdmemberand16.5percentoftheoccupiedreturnswerecompletedbyaproxy. %>!% ЀTherewere1,255,579continuationformsusedinNRFU.Approximately93.6percentofthe ' #' returnshadonecontinuationformattached,approximately2.9percentofthereturnshadtwo (#( continuationformsattached,andfewerthan1percentofthereturnshadthreeormore )$) continuationformsattached.Thenumberofcontinuationformsattachedrangedfromoneform l*%* toasmanyas99forms. R+&+  8,', NRFUenumeratedahigherpercentageofmultiunitsandrentedunitsthanwereself  enumerated.NRFUalsoenumeratedahigherpercentageofmales,youngpeople,Hispanics,and  peopleofallracesexceptWhites. | Therewere688,944addressesaddedand6,023,232addressesdeletedduringNRFU.The H majorityoftheaddedanddeletedaddressesweresingleunitsinthemailout/mailbackareas.The . ~ addsanddeletesweremostlycompletecitystyleaddresses.  d Recommendationsinclude:  0 9XX4, XX, X9" i"   ioG23  0    MonitorthefollowupworkloadinrealtimetoreducethenumberofNRFUcaseswith    unknownpopulationcountsandthenumberoflostNRFUenumeratorreturns. ioGG݌ (#(# Ќ  " i"   iH23  0    PeriodicallyidentifyandremovelatemailreturnsfromtheNRFUworkloadtoreducethe ^  NRFUworkloadandthenumberofhousingunitswithmultipledatacaptures. iHI݌D (#(# Ќ  " i"   iEJ23  0    Implementasufficientqualityassurance(QA)programtoensuretheaccuracyofthe ` NRFUproductionfilesandtheproperuseofenumerationtechniques. iEJrJ݌F(#(# Ќ  " i"   iK23  0    Developstandards/benchmarkswithwhichtomeasure/judgetheresults. iKK݌(#(# Ќ  EvaluationH.7,NonresponseFollowup(NRFU)EnumeratorTraining (BurtandMangaroo  2003) t ThesuccessoftheNRFUoperationwashighlydependentontheCensusBureausabilityto @ quicklydevelopskilledemployeeswhowereabletoeffectivelyperformthetasksofNRFU &v enumeration.DuringCensus2000,theCensusBureauhiredmorethanonehalfmillion  \ temporaryworkerstoconducttheNRFUoperation.Thisevaluationexaminedtheeffectiveness B oftheCensus2000NRFUenumeratortrainingprogram.TheevaluationusedtheKirkpatrick ( trainingassessmentmodeltoevaluatethetraineessatisfactionwiththetrainingprogram,their  knowledgefollowingtraining,andtheironthejobperformance.* '  2      ׀Themethodologyincludeda  contentreviewofthetrainingmaterials,observationreportsontrainingdeliveryandNRFU   enumeration,andsurveysanddebriefingsofenumeratorsandcrewleaders. p!  AbouthalfoftherecommendationsfromtheCensus2000DressRehearsalevaluationwere <#" incorporatedintothe2000trainingpackage,eithercompletelyorpartially.The "$r# recommendationsthatwereincorporatedcontributedtoanimprovedtrainingprogram.The %X $ recommendationsthatwerenotincorporateddidnotseemtosignificantlyimpactthe %>!% effectivenessofthetraininginpreparingtheenumeratorstocollectCensusinformation. &$"& _ (#( Mostenumeratorswereknowledgeableofcensusconcepts.Withtheexceptionofreading  questionsasworded,enumeratorsconsistentlydemonstratedeffectiveinterviewingskills.  EvaluationresultsindicatedthatthetrainingpreparedtheNRFUenumeratorstoeffectively b performtheirjobandtocarryoutthetaskstheyweretrainedtodo.Almostallofthe H enumeratorsdisplayedtheirCensusIDateachhousehold,properlyidentifiedthemselvesandthe . ~ purposeoftheirvisit,anddeterminedCensusDayresidencystatus.Mostconsistently  d confirmedthattheywereatthecorrectaddressandprovidedaPrivacyActNotice.Themajority  J ofNRFUenumeratorsrecordedanswersaccuratelyandlegibly.AsizableproportionofNRFU  0 enumerators,however,didnotalwaysreadthequestionsexactlyaswordedandfrequentlydid    notusetheflashcardsprovided.Thetwomajorareasinwhichenumeratorsseemedlesslikelyto    followprocedureswereaskingaboutHispanicoriginandrace.   Overall,enumeratorsweresatisfiedwiththeamountandcontentofthetrainingtheyreceived. ^  TheevaluationresultsindicatedthattheCensus2000NRFUtrainingprogramwaswellreceived *z  bytraineesandproducedenumeratorswhocouldeffectivelycollectneededCensusdata.The ` resultsalsosuggestedsomeareastheCensusBureaucouldfocusonindevelopingNRFU F enumeratortrainingfor2010. , " i"   i?Z23  0    Increasethetrainingtimeallottedtoareasinwhichenumeratorsfeedbackindicatedthey  feltlesswellprepared,withparticularemphasisoninteractingwithreluctantrespondents  andrefusals. i?ZlZ݌t(#(# Ќ  " i"   i[23  0    Continuetoplaceemphasisonreadingallofthequestionsexactlyasworded,adding @ additionalexplanationsonwhyreadingquestionsverbatimissoimportanttodata &v quality. i[\݌ \(#(# Ќ  " i"   is]23  0    Conductdebriefingsofenumeratorandcrewleadersinthe2004CensusTesttoget ( insightonhowtoimprovetheuseofflashcardsandotherjobaidsandonhowtomake  improvementsonpreparingenumeratorstoasktheethnicityandracequestionsexactly  asworded. is]]݌ (#(# Ќ  " i"   in_23  0    Continuetopr_ovide_ԀanopportunityforthefieldworkcomponentofNRFUtrainingand p!  enforceinclusionoffieldworkinalltrainingsessions. in__݌V"!(#(# Ќ  " i"   i`23  0    RestructuretheNRFUenumeratormanualtobemoreconsistentwiththetrainingguide "$r# intermsoforganizationandcontent. i`a݌%X $(#(# Ќ  " i"   i.b23  0    3,4XX4, X3Addtothetrainingmaterialsa FrequentlyAskedQuestionsjobaid,outlining &$"& potentiallydifficult_questions_Ԁthatrespondentsmightaskandappropriateresponses. i.b[b݌' #'(#(# Ќ  " i"  ',4,4' ic23  0    Useadditionalmediasuchasaudiotapes,videos,flipcharts,posters,andslidesin )$) trainingandassuretheyareused. icLd݌l*%*(#(# Ќ   8,', " i"   iwe23  0    Increasetheuseofroleplaying,varyingthesituationstoincludereluctantrespondents  andrefusals. iwee݌(#(# Ќ  " i "   if23  0    Useatrainerandtrainingassistanttomodelinterviewingskills,techniques,andstyles. iff݌b(#(# Ќ  EvaluationH.8,OperationalAnalysisofEnumerationofPuertoRico (McNally2003) . ~ Census2000wasthefirsttimethatanU/Lmailbackmethodologywasusedtoconductthe  J enumerationinPuertoRico.StatesideU/Loccurredinareasthatweredeterminedtocontain  0 someproportionofRuralRoute,P.O.Box,orothernoncitystyleaddresses.Theseareaswere    primarilyrural,butnottooremoteorsparselypopulated.InPuertoRico,U/Lwastheonlytype    ofenumeration.   DuringtheU/Loperation,questionnaireswithpreprintedaddresslabelswerehand-deliveredto ^  everyhousingunitontheaddresslist.Existinghousingunitsthatwerenotlistedontheaddress D  registeralsorequiredquestionnaires,butthesequestionnaireswerehand-addressedandthe *z  addresseswereaddedtotheaddressregister.Staffdeliveringthequestionnairesalsomade ` corrections,deletions,andadditionstotheaddresslistsandmaps. F Workingwithmapsandmapspotswasreportedasthemostchallengingsituationduetousing  ruralproceduresinanurbanlocation.Otherproblemsencounteredweretrainingmanuals  arrivedlate,merginglongandshortformquestionnaireswasaverytimeconsumingprocess,  andthemapsforNRFUandCIFUwerenotupdatedfrompreviousoperations. t Therewerealmost1.5millionaddressesinPuertoRico.Thisnumberrepresentshowmany @ addresseshadeitheralabeledquestionnairethatwastobedeliveredduringU/Lorahand &v addressedquestionnaireforaunitthatwasaddedtotheaddresslistduringtheU/Loperation.  \ Someoftheaddressesontheaddresslistweredeletedasnonexistentornonresidentialduring B theU/Loperation.Thisisincludedaspartoftheworkloadbecausetheeffortrequiredtotryto ( locatesuchhousingunitsisacomponentoftheoperation.  Addsaccountedfor7.6percentofthew_orkload_ԀinPuertoRico.Ofthe111,787U/Laddsin   PuertoRico,83.7percentwereincludedinthefinalcounts.Deletesaccountedforalmost8.4 p!  percentoftheworkloadinPuertoRico. V"! AsofApril18,2000theresponserateforPuertoRicowas48.4percentcomparedto59.3 "$r# percentforstateside.SixtyfourpercentofthehouseholdsinPuertoRicoreturnedtheirCensus %X $ 2000questionnairebymail,alowreturnratewhencomparedwiththestatesiderateof77.9 %>!% percent.Manyquestionnairesthatweremailedbackwerenotrecordedinthesystemand &$"& becamepartoftheNRFUworkload.Thisledtomultiple,unnecessaryvisitstohouseholds. ' #' XzXXXzBasedontheCensus2000experience,recommendationsinclude: )$)  * #$%&'((xx!* #$%&'(" i"  !,4! it23  0    BuildonexperiencefromCensus2000andtheACSinPuertoRicotousethe R+&+ _mailout_/mailbackdatacollectionmethodologyforatleastpartofPuertoRico. it#u݌8,',(#(# Ќ  " i"   i{v23  0    Conductfurtherresearchintowaystoimprovecensusmapstomakethemeasierfor  enumeratorstouseandtoimprovetheaccuracyofmapspotting. i{vv݌(#(# Ќ  " i"   iw23  0    Ensurethattrainingmaterialsandtrainingmanualsarriveontime.#XzXXXzs# iwx݌b(#(# Ќ  EvaluationH.9,LocalCensusOfficeProfileforCensus2000 (Imel2003) . ~ TheLCOProfileforCensus2000isadatabaseofawidevarietyofvariablesaggregatedatthe  J LCOlevel.DatawereproducedspecificallyfortheLCOdatabase,whichalsocontainsdata  0 fromCensus2000evaluations.TheLCOprofilecontainsdatafromthefollowingtypesoffiles    andsystems:addressframefiles,dataprocessingfiles,andfieldtrackingsystems.Itcontains    morethan1400statisticsforeachofthe520LCOs.Thesedataincludetotalhousingcounts,   occupiedhousingcounts,vacanthousingcounts,totalpopulationcounts,responserates,various x  operationsummaries,aswellasmanydemographiccharacteristics.Thedatabaseisintendedto ^  provideinternalCensusBureauresearchersandplannersarich,comprehensivedatasetwhich D  willaddahistoricalperspectiveandprovidebenchmarksforplanningthe2010Census. *z   EvaluationH.10,DateofReferenceforAgeandBirthDateusedbyRespondentsofCensus F 2000 (Carter2002b) , ThisevaluationexaminedhowwellrespondentsansweredthecensusasofCensusDay,April1,  2000,bylookingathowrespondentsansweredtheageanddateofbirthquestion.Theway  respondentsanswerthisquestioncanbeinfluencedbywhetherornottheyareusingCensusDay t astheirdateofreference. Z Astatesreturnrateseemstobecorrelatedwiththedateofreferenceforthatstate.Asthereturn &v rateincreases,thedateofreferenceforthestateisclosertoApril1,2000.Ahigherreturnrate  \ inastatemeansmorerespondentsarereturningtheirquestionnairesthroughthemail.Itisalso B verylikelythattheserespondentswillnotbepartofNRFUandtheyareenumeratedcloserto ( April1,thuslesslikelytomisreporttheirage.Ifthereturnrateislow,thatwouldmeanahigher  percentageofpeoplearebeingenumeratedinNRFU.NRFUtakesplaceatalaterdate,sothe  respondentsenumeratedinNRFUseemtohaveagreaterpropensitytouseadateotherthan   CensusDaytoreporttheirage. p!  Theanalysisshowedthat89.8percentofpersonshadtheirreportedageconsistentwiththeir <#" calculatedage.Therewere1.8percentthatunderreportedtheiragebyoneyearand6.0percent "$r# thatoverreportedtheiragebyoneyear.Thesepeoplemayhavepotentiallymisreportedtheir %X $ ageduetousingsomedateother_than_ԀApril1,2000asthedateofreferencewhenreportingtheir %>!% age.Theremaining2.4percentmisreportedtheiragebymorethanoneyear,whichmeansthe &$"& misreportingcanonlybeattributedtosimplereportingerror. ' #' Thereweretwosituationswhererespondentsmayhavehadproblemsreportingagecorrectly; )$) thedateofcheckinoftheformwiththepersonsdatawasbeforethepersonsbirthdayandboth l*%* werebeforeApril1,andthedateofcheckinoftheformwiththepersonsdatawasafterthe R+&+ personsbirthdayandbothwereafterApril1.Inthefirstsituation,10.3percentofthepersons 8,', inthiscategoryunderreportedtheirage.Inthesecondsituation,40.1percentofthepersonsin  thiscategoryoverreportedtheirage.Thesepercentagesarehigherthananypercentobservedin  anyoftheothersituationsforthattypeofmisreporting.Thismeansthatthetimeatwhicha | personisrespondingtothecensusdoesaffecthowheorshereportsage. b Therewere80.4percentofhouseholdsthathadeverypersoninthemwiththeagecorrectly . ~ reported.Thiscomparesto89.8percentofpersonswithagecorrectlyreported.  d ThecensusmomentoraveragedateofreferencemovedfromMay5in1990toApril20in  0 2000.Thisimprovementmaybeduetothechangeinquestionnairedesignandinthe    enumerationtimeframe.The2010Censusquestionnaireshouldstressthattherespondentsare    toprovidetheirageasofCensusDay,April1,asinCensus2000.Thiswillhelprespondents   notmisreportage.Also,acompressedCensusenumerationtimeframemayaidrespondentsto x  correctlyreportage. ^  Respondentsenumeratedbypersonalvisittendedtooverreportage.Enumeratorsshouldhave *z  thisproblemexplainedtothemandtrainingshouldstresstheimportanceofCensusDayasthe ` referencedate.EnumeratorsshouldknowthatrespondentsneedtohearApril1,2010,sothey F cancorrectlyprovidetheirinformation. , Theproblemsthatareobservedinagereportingrevealedproblemswithrespondentsreferencing  April1whenprovidingagedate.Thisissomewhattrivialbecauseagecanbecalculatedfrom  dateofbirth.ThereareotherissuesthataresensitivetotheApril1referenceday,suchas t ResidencyRules,thatcannotbecorrected. Z  CategoryICoverageImprovement   \ SummariesofthefiveevaluationsintheCoverageImprovementcategoryfollow. (  EvaluationI.1,CoverageEditFollowup (Sheppard2003)  TheCEFUoperationforCensus2000wasusedtoincreasewithinhouseholdcoverageand p!  improvedataqualitybycollectingpersondataforallpersonsbeyondthefirstsixinlarge V"! householdsandbyresolvingcountdiscrepanciesbetweenthereportedhouseholdpopulation <#" countandtheactualnumberofdatadefinedpersonsrecordedonthecensusform.Thepurpose "$r# ofthisevaluationwastodocumenttheCEFUoperation. %X $ Whilethecoverageeditfailureworkloadwasprojectedtobeover3.1million,theactual &$"& workloadincluded2,544,072CEFUcasesfromCensus2000mailbackandInternetforms. ' #' Largehouseholdcasesmadeupalmost55percentoftheCEFUcases(1,395,623).Theedit (#( failurerateforlargehouseholdcaseswas1.7percent.Countdiscrepancycasesmadeuptherest )$) (1,148,449).Theeditfailurerateforcountdiscrepancycaseswas1.4percent.Justover60 l*%* percent(699,379cases)ofthecountdiscrepancycaseswereselectedbecausethenumberofdata R+&+ definedpersonsontheformexceededtherespondentreportedhouseholdsize.Therestofthe 8,', countdiscrepancycaseswereselectedbecausethenumberofdatadefinedpersonsontheform  waslessthantherespondentreportedhouseholdsize(449,070cases).  Therewere1,251,971casescompletedduringtheCEFUoperation.Thiswas53.5percentofall b theeligibleandattemptedcases.TheCensusBureauweremoresuccessfulcompletinglarge H householdcases(57.4percent)thancountdiscrepancycases(48.5percent).Thelargestreason . ~ forincompletecaseswasinabilitytocontacttherespondentbytelephone.TheCensusBureau  d madetwoattemptstoobtaintelephonenumbersforcaseswhereonewasnotcorrectorpresent  J onthemailbackform.However,only21.1percentofthecaseswithchangedtelephonenumbers  0 werecompleted.Sincetherewasnofieldfollowup,theCensusBureauwasunabletocomplete    anyofthecaseswherevalidtelephonenumberscouldnotbeobtained.Theserepresented24.0    percentofalleligibleCEFUcases.   TheCEFUinstrumentwaseffectiveinitstwomainobjectives:correctingincorrectrostersand ^  collectingpersondata.TheCEFUoperationsuccessfullyimprovedcoverageanddecreasedthe D  differentialundercount.In232,777cases,or18.6percentofallcompletedcoverageeditcases, *z  oneormorepersonswereadded,deleted,orremovedasaduplicate.Atotalof410,565persons ` wereadded,deleted,ormarkedasduplicatestocorrecttherosterofahousehold.The152,683 F personswhowereaddedtothehouseholdrosterweremorelikelytobemembersoftraditionally , undercountedpopulationsthanpersonsintheoverallpopulationenumeratedinCensus2000.  Thesepersonsweremuchmorelikelytobeunder24,beofaraceotherthanWhite,andtobe  Hispanic.Theywereslightlymorelikelytobe65yearsoldorolder,bemale,andhavethe  householderbeanowner.Therewere257,882personswhoweredeletedorremovedas t duplicatesfromthehouseholdrosterduringtheoperation.Thesepersonsweremuchmorelikely Z tobebetween15and24orover65yearsoldandtobeBlackthanpersonsintheoverall @ populationenumeratedinCensus2000.TheywereslightlymorelikelytobeofHispanicorigin, &v befemale,andhavethehouseholderbeanowner.  \ TheCEFUoperationactuallyresultedinanetlossof105,199personscomparedtotheoriginally ( completedCensus2000selfresponseforms.However,whilethenetimprovementtothecensus  fromCEFUoperationwasadecreaseinthepopulation,itdidimprovetheaccuracyofCensus  2000.Throughtheprobinginterview,theCEFUincreasedthelikelihoodthatthe410,565   peoplewhowereadded,deleted,ormarkedasduplicateswerecountedinthecorrecthousehold. p!  Giventheresultsandlimitationsofthedata,thefollowingaresomerecommendationsforthe <#" CEFUoperationinCensus2010: "$r#  * #$%&'((xx* #$%&'(" i"   i23  0    ContinuetoconductaCEFUoperationinfuturecensuses.Includecountdiscrepancy %>!% casesandlargehouseholdcases,aswellasothercasesidentifiedashavingasignificant &$"& possibilityofcoverageproblems. iԣ݌' #'(#(# Ќ  " i"   iV23  0    DevelopwaystoincreasethecompletionrateforCEFUoperations. iV݌)$)(#(# Ќ   8,', " i"   id23  0    Improvecasefilecreation,management,softwaretesting,andtransmittalproceduresof  inputandoutputfilestoavoidlossofdataandtoensureinformationisavailableto  conductinterviewsasplanned. id݌|(#(# Ќ  " i"   i23  0    ImprovethedesignoftheCEFUinstrumenttoimproveeffectivenessandreduce H respondentburden. i>݌. ~(#(# Ќ  " i"   i=23  0    Collectevaluationdatainfuturetestsofcoveragefollowupoperationstohelpimprove  J themethodologyusedtoconductfollowupinterviews! !. i=j݌ 0(#(# Ќ  " i"   i23  0    AssignthefinalhouseholdsizeforcountdiscrepancycasesnotcompletedduringCEFU    byassumingthatthecasesthatarenotcompletedbehavesimilarlytothosethatare   completed. i݌x (#(# Ќ  EvaluationI.2,EvaluationofNonresponseFollowupWholeHouseholdUsualHome D  ElsewhereProbe (ViatorandAlberti2003) *z  TheWholeHouseholdUsualHomeElsewhere(WHUHE)probewasaquestionnairecoverage F improvementusedtodetermineifallmembersofahouseholdonthedayoftheinterviewhad , anotherresidencewheretheylivedmostofthetime(theirCensusDayaddress).Thisprobewas  accomplishedbyimplementingasetofscreeningquestionsontheSimplifiedEnumerator  Questionnaire.Incaseswhereahouseholdindicatedthatallhouseholdmembershadanother  residencewheretheylivedmostofthetime,enumeratorscompletedablank,unlabeled t SimplifiedEnumeratorQuestionnairefortheirCensusDayaddressor usualresidence.This Z questionnairewasusedtoensureacompleteandaccurateenumerationattheaddressoftheusual @ residence. &v Atotalof151,775questionnaireswerecompletedforWHUHEhouseholdsfortheirusualplace B ofresidence.Ofthesereturns,58,027matchedtoanexistingaddressontheDMAF.Another ( 55,286returnsweregeocodedbutnotmatchedtoanexistingaddress.Ofthese55,286returns,  only606weregeocodedintimetobesenttotheFieldVerificationoperation.Mostofthe  remaining54,680returnswereaddedtothecensusbutnotincludedintheFieldVerification   operation.Forthe606returnsthatweregeocodedandsenttoFieldVerification,273were p!  verified,271weredeleted,59wereduplicates,andresultswerenotreportedforthreereturns. V"! Finally,38,462returnscouldnotbegeocodedormatchedtoanexistingaddress. <#" Therewere55,987WHUHEaddressesthatwereenumeratedbyanotherprocedure.Morethan %X $ 14percentofthesewerereportedtobevacantornonexistent(delete)onCensusDay.There %>!% were54,915addressesthatwerenotenumeratedbyanotherprocedure.Therespondentfor &$"& nearly62percentoftheseaddresseswasaneighbororotherproxycomparedwithonlyabout32 ' #' percentforaddressesenumeratedbyanotherprocedure.Morethan71percentoftheaddresses (#( notenumeratedbyanotherprocedurewerereportedasvacant.Formorethan76percentofthese )$) vacantaddressestherespondentwasaneighbororotherproxy. l*%*   8,', Therewere113,991datadefinedpersonsenumeratedonoccupiedWHUHEforms.Ofthese  persons,75,254werefoundonothercensusreturnsattheaddressand38,737werenotfoundon  othercensusreturnsattheaddress.Ofthe38,737personswhowerenotfoundonotherreturns, | 29,302wereselectedbythePrimarySelectionAlgorithmforinclusioninthecensus.Theseare b personswhowerenotenumeratedbyotheroperationsandwereaddedtothecensusbythe H WHUHEprogram. . ~ Thereappearedtobeconsiderableconfusionamongenumeratorsconcerninghowtoenumerate  J theWHUHEhouseholds.Manyoftheusualresidencesforthesehouseholdswerereportedas  0 vacantordeletedhousingunits.Thisiscontrarytotheconceptofausualhomeelsewherefor    theCensusDayhousehold.Manyofthequestionnairescompletedfortheusualhomeofthe    CensusDayhouseholdwerecompletedbyarespondentwhoisnotamemberofthehousehold.   Anecdotalevidenceshowsmanyinstancesofmultiplequestionnairesfilledbythesameproxy x  respondent. ^  Frequently,thereweremissingandinconsistentdataforresponsestotheintroductoryquestions *z  usedtoidentifyWHUHEhouseholds.Becausetherewasnowaytolinkacensusreturnforthe ` addressonthedayoftheinterviewwiththereturnfortheWHUHEaddress,themissingand F inconsistentresponsespreventedtheaccurateidentificationofresponsesthatshouldhave , generatedaWHUHEreturn.  Recommendationsinclude:  " i"   iY23  0    TakeintoconsiderationthesmallnumberofpersonsaddedtothecensusbytheWHUHE Z probeinjudgingthepotentialeffectivenessofthisprogramforthe2010Census. iY݌@(#(# Ќ  " i"   iʿ23  0    Researchwhetherornotitisasoundpracticetoaddrespondentprovidedaddresses,such  \ astheWHUHEaddresses,withoutverifyingthemthroughaFieldVerificationoperation. B  iʿ݌ (#(# Ќ  " i"   iQ23  0    CoverthepurposeofandtheproceduresfortheWHUHEprogramthoroughlyin  enumeratortrainingtoensureabetterunderstandingoftheprogramandhigherquality  data. iQ~݌ (#(# Ќ  " i"   i23  0    Addressthetreatmentofproxyresponsesaboutusualhomeelsewhereinformationin V"! futurecensuses. i݌<#"(#(# Ќ  " i"   i23  0    Incorporateeditsintofutureinteractiveelectronicenumerationdevicestodetectand %X $ correctinconsistentdataproblemsasthedataarebeingcollected. i4݌%>!%(#(# Ќ  " i"   im23  0    CreateamechanismtolinkeachWHUHEreturnwiththecensusreturnthatgeneratedit ' #' todesignnewQAprocesses. im݌(#((#(# Ќ   l*%*  EvaluationI.3,EvaluationofNonresponseFollowupMoverProbe (Keathley2004)  Allenumeratorquestionnairescontainedamoverprobe.Themoverprobeallowedenumerators | toidentifyhouseholdsthatmovedintoNRFUorCIFUhousingunitsafterApril1,2000anddid b notreturnacensusquestionnairefortheircensusdayaddress(nonresponseinmover H households).Enumeratorsinbothoperationswouldthenattempttocompleteaseparate . ~ enumeratorquestionnaireforeverynonresponseinmoverhouseholdfortheaddressatwhichit  d livedoncensusday.Theseenumeratorquestionnairesrepresentedthenonresponseinmover  J householdscompletedcensusquestionnaires.Thepurposeofthisevaluationwastodetermine  0 theusefulnessofthemoverprobe.    Ofthe105,480,101occupiedhousingunitsintheU.S.(the50statesandDC)enumeratedin   Census2000,22,850wouldnothavebeenenumeratedwithoutthemoverprobe.Thisrepresents x  0.02percentofthetotalU.S.occupiedhousingunitcount. ^  Therewereatotalof45,507,823enumeratorquestionnairesfromthetwofollowupoperations. *z  The22,850enumeratorquestionnairesrepresentingoccupiedhousingunitsenumeratedonlyby ` themoverproberepresents0.05percentofallNRFUandCIFUquestionnaires. F Therewere125,585enumeratorquestionnairesthathadinformationindicatingthatthey  representednonresponseinmoverhouseholds;only18.19percent(22,850)oftheserepresented  householdsthatCensus2000wouldnothaveotherwiseenumeratedwithoutthemoverprobe.  Recommendationsinclude: Z " i"   id23  0    Reevaluatethemoverprobeinfuturecensusandtestcensusoperations. id݌&v(#(# Ќ  " i"   ic23  0     @ Usingthemoverprobeimprovedtheaccuracyofthehousingunitcount(andother B relatedstatistics). ic݌((#(# Ќ  " i "   i23  0    Thecostofkeepingthemoverprobeinfuturecensusoperationsmightbenegligible. i݌(#(# Ќ  " i "   i23  0    Computerizedversionsoffutureenumeratorquestionnairescouldimprovethe p!  effectivenessofthemoverprobe. i݌V"!(#(# Ќ  " i "   i23  0    Developawaytolinkseparateenumeratorquestionnairesforthesamefollowup "$r# operationhousingunit(oneenumeratorquestionnairerepresentingthefollowup %X $ operationhousingunit,theotherrepresentingthehousingunitatwhichanonresponse %>!% inmoverhouseholdresidedoncensusday). i݌&$"&(#(# Ќ   (#(  EvaluationI.4,CoverageImprovementFollowup (Moul2003)  CoverageImprovementFollowup(CIFU)wasdesignedtoimprovecoverageofhousingunitsin | the_mailout_/mailback,U/L,andUrbanU/Lareas.Therewere121,894,831housingunitsinthese b mailbackareasthatwerepotentiallyeligibleforCIFU.Theworkload(includingPuertoRico) H consistedof8,854,304housingunits.Mostofthisworkloadconsistedofunitsclassifiedas . ~ vacantordeleteinNRFU.Theworkloadalsoincludedunitsthatwereidentifiedasvacantor  d deleteintwocensusoperations,unitsidentifiedasseasonalvacants,unitsidentifiedas_UAA_,  J addsfromthenewconstructionoperation,addsfromtheU/LandUrbanU/Loperations,blank  0 mailreturns,lostmailreturns,nonrespondentsinPanels7,8,and9oftheResponseModeand    IncentiveExperiment,February2000andApril2000DSFadds,addsfromtheLUCA1998and    1999Appealsprocess,Hialeah,FloridaNRFUunits,POP99s(housingunitsidentifiedas   occupiedduringNRFUthatdidnothaveapopulationcount),andResidualNRFUunits. x  TheCIFUoperationwasconductedinthreeseparatewavesasgroupsofLCOscompleted D  NRFU.Wave1included342LCOsandbeganonJune26andfinishedonJuly26.Wave2 *z  included175LCOsandbeganonJuly10andfinishedonAugust10.Wave3includedthree ` LCOsandbeganonJuly30andfinishedonAugust23. F ThisevaluationdevelopedaprofileoftheCIFUunitstoprovideCensusManagerswithcritical  informationneededforplanningthe2010Census.  TheCIFUoperationfollowedup3.9millionvacantunitsand2.6milliondeleteunits. t Approximately21.9percentofthevacantunitswereconvertedtooccupiedand24.6percentof Z thedeleteswereconvertedtooccupied.Theseconvertedunitsresultedinanetgainof @ approximately3.1millionpeople.Approximately18.1percentofthedeleteswereconvertedto &v vacant.  \ Inthe1990fieldfollowupoperationVacant/Delete/MoversCheck,approximately8.7percentof ( thevacantswereconvertedtooccupiedand6.4percentofthedeleteswereconvertedto  occupied;approximately5.3percentofthedeleteswereconvertedtovacant.Comparedto  Census2000,the1990vacantanddeleteworkloadswerelargerandtheconversionrateswere   lower.Thesedifferencesweretheresultofchangesintheuniverserulesforinclusion(i.e.,there p!  weredifferentrulesforincluding/excludingvacantanddeletedunits). V"! AttheendofCIFU,approximately26.8percentoftheunitswereoccupied,43.4percentwere "$r# vacant,and29.7percentweredeletesandonly542ofthe8.9millionhousingunitshadan %X $ undeterminedstatusattheendoftheoperation.Morethan88percentofthelostmailreturns %>!% and81.2percentoftheblankmailreturnsyieldedvalidhousingunits.Approximately52.9 &$"& percentofthenewconstructionaddsand58.5percentoftheDSFaddsweredeletedand ' #' approximately63.6percentoftheLUCAAppealsaddswereultimatelydeleted,whichconfirms (#( thefindingsinearlieroperationsthattheseaddresseswerenotvalidaddresses. )$)   R+&+ Therewere5.3millionpeopleenumeratedinCIFU.LiketheNRFUoperation,CIFUwas  successfulinenumeratingahigherpercentageofthegroupsthataretypicallyundercounted:  males,youngpeople(34yearsoldandyounger),Hispanics,_Blacks,_ԀandSomeOtherRace. | Recommendationsinclude: H " i "   i23  0    ContinuetoimprovecoveragebyfollowingupvacantanddeletedunitsfromNRFU. i݌ d(#(# Ќ  " i "   i23  0    Continuetofollowupanylostorblankmailreturns.Consideraddinga vacantoption  0 tothemailbackquestionnairesothatrespondentscouldindicatetheunitwasvacanton    CensusDaysothatvaluableresourcesarenotwastedfollowinguplegitimateblank    returns. i ݌ (#(# Ќ  " i"   i23  0    Investigatewaystoimprove/screenthedatareceivedfromlocalgovernmentstoavoid ^  spendingtimeandmoneyfollowingupinvalid/baddata. i݌D (#(# Ќ  EvaluationI.5,CoverageGainsFromCoverageQuestionsC1andC2on ` EnumeratorCompletedQuestionnairesforCensus2000 (ZelenakandNguyen2003) F Thefocusofthisevaluationwastheuseandeffectivenessofcoveragequestionsonenumerator  completedquestionnairesforCensus2000.Theintentofthesequestionswastoidentifypeople  whootherwisewouldhavebeenmissedorincludedinerror.Thecensusoperationswhichused  thesequestionswereL/E,U/E,NRFU,andCIFU.Enumeratorsbeganbyaskinghowmany t peoplewerelivingorstayinginthehousingunitonCensusDay.Aftercollectingthe Z appropriatepersonandhousingunitinformation,theenumeratoraskedtwocoveragequestions, @ whichweredesignedtogetanaccurateenumerationofallpeopleandhousingunits. &v Forthefirstquestion,C1,theenumeratorreferredtoCensusDay,April1,2000,andasked: B IneedtomakesureIhavecountedeveryonewholivedorstayedhereonApril1,2000.DidI  miss  Єanychildren,includingfosterchildren?   Єanyoneawayonbusinessorvacation? p!  Єanyroomersorhousemates? V"! Єanyoneelsewhohadnootherhome? <#" Forthesecondquestion,C2,theenumeratorreferredtoCensusDay,April1,2000,andasked: %X $ TheCensusBureauhasalreadycountedcertainpeoplesoIdontwanttocountthemagain &$"& here.OnApril1,2000,wereanyofthepeopleyoutoldmeabout ' #' Єawayatcollege? (#( ЄawayintheArmedForces? )$) Єinanursinghome? l*%* Єinacorrectionalfacility? R+&+  8,', _Enumeratorsweresupposedtomarkeither Yesor Noforeachcoveragequestionandthen  takeappropriateactionwhenevertheresponsewas Yes.However,aboutonethirdofthetime,  enumeratorsleftthesequestionsblank.Approximately1.1percentoftheresponseswere Yes | forC1,meaningthatsomeonehadbeenmissed,and0.7percentwere YesforC2,meaning b thatsomeoneshouldbecountedelsewhere.Amongtheoperations,theCIFUoperationsawthe H lowestpercentageofreturnswiththesequestionsmarked,49.4percentand49.2percentforC1 . ~ andC2,respectively.TheresponserateforthesequestionsintheNRFUoperationwas  d approximately68percentforboth.  J Inconsistenciesappearedforthepeopleassociatedwiththesereturns.Only21.8percentofthe    returnsthathadthe YesboxmarkedforC1,hadatleastonepersonadded.Only43.4percent    ofthereturnswithC2markedas Yeshadatleastonepersondeleted.   Lackofinformationmadeitdifficulttogetanaccurateaccountofthepeoplewhoweremissed ^  orincludedinerror.Therefore,anetresultofpeopleaddedordeletedmaynotbeinferredfrom D  thedatainthisreport.Forthesamereason,thedemographicdataincludedinthisdocument *z  maynotaccuratelyreflectthedistributionsofthepeoplewhoweretrulyaddedtoordeleted ` fromthehouseholdrosters. F Basedontheinformationaboutthenumberof Addand Cancelboxesmarked,theCensus  Bureauadded77,050peopleanddeleted83,160people.Amongthepeoplerecordedasadds,  46.6percentwerearaceotherthanWhite,57.9percentwereyoungpeople(ages0to24),56.5  percentweremales,and51.2percentwererenters.Thesegroupsaretraditionallyundercounted, t however,theCensusBureaucannotinferthatthesetwocoveragequestionsaregoodfor Z improvingthedifferentialundercountbecauseoftheinadequacyofthedatacollected. @ Recommendationsinclude:  \ " i"   i23  0    Provideadditionalspaceimmediatelyfollowingthecoveragequestions(forenteringthe ( namesofthepeopletobeaddedordeleted)toimprovethepercentageofreturnswiththe   Addand Cancelboxesmarkedwhenthecoveragequestionsareansweredas Yes. i݌(#(# Ќ  " i"   i23  0    Useautomatedinstrumentsandhavemoretrainingforenumerators(onthepurposeof p!  askingthesequestions)toimprovethedatacollectionprocess. i݌V"!(#(# Ќ  CategoryJEthnographicStudies  %X $ SummariesofthethreeevaluationsintheEthnographicStudiescategoryfollow. &$"&  EvaluationJ.2,EthnographicSocialNetworkTracingofHighlyMobilePeople (Brownrigg (#( 2003) )$) Residentialmobilityhaslongbeenidentifiedasabehaviorthatchallengesaccurateenumeration R+&+ andcoverage.Tolearnmoreabouthowresidentialmobilityimpactscensuscoverage,the 8,', EthnographicSocialNetworkTracingProjectresearchedsocialnetworkswhichincludehighly  mobilepeople.Highlymobilepeopleweredefinedaspeoplewhomakeresidentialmovesmore  oftenthanmostpeopleintheU.S.orwhohabituallymigrateamongdomiciles.Socialnetworks | wereformallydefinedandmodeledbyobservingpeopleinteractoverasixmonthperiod. b Researcherstracedparticipantsinteractinginthesocialnetworkstotheaddressesandlocations H oftheirdomicilesandreportedtheidentitiesandcharacteristicsofparticipants,setsofco . ~ residents,andthedomicilestheyoccupied.  d Participantsinthesixsocialnetworksresearchedwereinvolvedindiversepatternsanddegrees  0 ofmobility:asocialnetworkofcamperswhocookedcommunally,includingsurvivalcampers    alongwithrecreationalcampers;seasonalworkerswhohabituallycirculatedamonganaverage    ofthreetermassignmentsatdifferentdistantworksites;afolkloricdancegroupmadeupof   MexicanformerfarmworkerssettlingintheruralMidwest;olderHaitianswhoworkedtogether x  inagriculturalfieldsintheSouth;commercialfishermen,theirfriendsandfamily,andindustry ^  employeeswhosocializedaroundaparticularAtlanticcoastfishingdock;andparticipantsina D  localchapterofanAmericanIndianmenssociety. *z  Variousassociationswerefoundbetweenthecharacterofindividualsmobility(whether F sedentary,residentiallymobile,orhabituallymobile),theirpositionsintheinteractingsocial , networksandmatricesofcoresidence,and"censusoutcomes."Inthesocialnetworkstraced,  feweroftheresidentiallyandhabituallymobileindividualswerefoundenumeratedinCensus  2000thanthosewhoremainedsedentary.Ifcensusoperationsdidnotlistorenumeratetheunit  thatwasthecensusresidenceofoneormoreindividuals,ordidnotplacetheunitinaccurate t censusgeography,thenitwasunlikelythatanycensusrecordscouldbefoundforanyoneliving Z inthatunit.Similarly,ifcensusoperationslistedandcorrectlyplacedacensusresidencein @ geography,butthendidnotenumerateitorenumerateditasvacantorwithentirelydifferent &v people,itwaslesslikelythatrecordsofanyofitscooccupantscouldbefound.Ifaunithad  \ beenlistedandenumeratedmorethanonce,thenallormostcoresidentsmightbeduplicated. B Theomissionorerroneousinclusionofcertainindividualsintheircorrectcensusresidence  whereatleastoneoftheirreportedcoresidentswasenumeratedandservedasthecensus  respondentarisefromrespondentbehaviors.These withinunitresultsreflectdynamics   amongcoresidentsthatinfluencewhoisreported. p!  Relationshipswerefoundinthesesmallscalesocialnetworkstudiesbetweenindividuals <#" mobilitybehaviorsandboth unitbasedand withinuniterrors.Habituallymobilepeople "$r# oftenstayedintypesofdomicilesCensus2000didnotlist.Inlistedhousing,howhousehold %X $ respondentsperceivedandinterpretedanindividualscurrentandhistoricmobilityinfluenced %>!% whetherornotthepersonwasreported.Theserelationshipscombinedtoproducetheneteffect &$"& thatmoreindividualstracedashabituallymobileorresidentiallymobilewereomittedthanwere ' #' foundenumerated. (#( _  l*%* Ifanindividualscensusresidencewasnotlisted,thenthatpersonhadnounitofenumeration  whereheorshecouldbecorrectlyenumeratedinCensus2000.Unlistedtypesincluded  unconventionaldomiciles(e.g.,campgrounds,dockedfishingboats,cheapmotels,farm | workers'rentallaborcamps),butalsoconventionalhousingunitssuchassinglefamilyhomes, b mobilehomes,townhouses,condos,andapartments. H Mostofthehabituallyandresidentiallymobilesocialnetworkparticipantswhowerefound  d enumeratedsharedcertaintraits:theyhadcensusresidencesinconventionalhousingand  J maintainedtieswithandrepeatedlyandroutinelyreturnedtothesamesetofresidentially  0 sedentarycoresidentsinonelocality.Aslongastheircensusresidencewaslistedand    enumerated,recordsforhabituallyandresidentiallymobilepeoplewithallthesetraitswere    found,nomatterhowoftenorhowfartheywentaway.Inthisresearchstudy,farmore   habituallyandresidentiallymobilepeoplelackedatleastoneofthetraitscitedaboveandwere x  omitted. ^   EvaluationJ.3,ComparativeEthnographicResearchonMobilePopulations (delaPuenteet *z  al2003) ` Thisevaluationprovidedresultsfromfourcommissionedethnographicstudiesconductedby , socialscientistswithextensiveknowledgeofspecificsubpopulationsthataretypically  residentiallymobile.Eachresearcherhadpreviouslyconductedresearchwithinhis/her  populationofinterestandwasknown,inmostcases,asatrustedindividualbythecommunity  he/shestudied.Thesefourethnographicstudiestookplacebefore,during,andafterCensus t 2000toevaluatethelifestylesofthegroupsandtoobserveresidentialmobilityactivitiesduring Z thesetimeperiods.Allresearchersusedacombinationofobservationandunstructured @ interviewsintheirfieldworks.Thefourtransientpopulationsexaminedintheethnographic &v studiesare:urbangangmembers,Irishtravelers,seasonalresidentsor"snowbirds"inArizona,  \ andAmericanIndiansresidingintheSanFranciscoBayArea.Acrossthesefourdistinct B populations,manycommonbarrierstoenumerationwerefound.Manyofthesebarriershave ( beenstudiedanddocumentedinpreviousethnographicstudiesofhardtoenumerate  populations.  Thebarrierstoenumerationidentifiedinthecurrentstudyinclude: p!   * #$%&'((xx* #$%&'(" i"   i23  0    Residentialmobility.Residentsmaybehardtocontact(i.e.,notreachedbytraditional <#" enumerationmethods!inpersonorbymail)ortheymayhavedifficultyprovidinga "$r# specificplaceofusualresidence.Mostindividualsinthestudywhowereawareofthe %X $ censusresidencerules,aspresentedonthecensusform,didnotfindthemhelpfulin %>!% reportingausualresidence. i݌&$"&(#(# Ќ   (#( " i"   i23  0    Distrustand/orfear.Therearetworelatedreasonswhythereisreluctancetoprovidethe  CensusBureauwithpersonalinformation.Thefirstisapplicabletopersonswhoengage  inillegalorunconventionalactivities.Thiscanrangefromviolationofacivilor | criminallawtoinvolvementinlivingarrangementsthatviolateeitherpublicorprivate b housingrules.Underlyingthisphenomenonisthefearthatinformationprovidedtothe H CensusBureauisnotkeptconfidentialbytheagencyandthatdivulgingsuchinformation . ~ mayresultinsomepenaltyorprosecutionifitfellintothewronghands.Thesecondand  d relatedreasonforthereluctancetoprovidepersonalinformationinthecensusisa  J broadersenseofdistrustingovernmentcoupledwiththeunwillingnesstoprovide  0 personalinformationtoanentitywhoseintentionsarequestioned. i݌  (#(# Ќ  " i"   iC$23  0    Irregularandcomplexhouseholdarrangements.Insomecases,violationofhousing   rulesanddistrustingovernmentmaypreventhonestresponses.Inothers,itisunclearto x  respondentswhomtoclassifyasahouseholdmemberwhensomeofthoselivinginthe ^  housearetransients. iC$p$݌D (#(# Ќ  " i"   iN&23  0    Disinterest.Insomecases,theCensusBureau'soutreacheffortdidnotresonateforsome ` mobilegroups.Eithertheywerenotexposedtothecampaignortheychosenottolisten F toitorbelievetheclaimsmadeinit.Thissegmentofthepopulationmayalsobe , unresponsivetomassmarketingstrategies.Consequently,theydonotunderstandwhy  thecensusisnecessarynordotheyunderstandtheprocess. iN&{&݌(#(# Ќ  Someofthekeyrecommendationsmadebytheethnographerswhoconductedthefieldworkare: t   " i"   ik)23  0    Enlistsupportfromcommunityorganizationstopromotecensusawarenessand @ encouragecensusparticipation. ik))݌&v(#(# Ќ  " i"   i*23  0    Directoutreachprogramstospecifictransientgroups. i**݌B(#(# Ќ  " i"   i+23  0    Clarifyresidencerulesfortransients. i++݌(#(# Ќ  " i"   im,23  0    Enumerateinnontraditionalsites. im,,݌ (#(# Ќ  " i"   iF-23  0    Makesurethatallundevelopedandpubliclandcampsitesaredesignatedfor V"! enumeration. iF-s-݌<#"(#(# Ќ  EvaluationJ.4,TheEnumerationofColoniasinCensus2000:Perspectivesof %X $ EthnographersandCensusEnumerators (delaPuenteandStemper2003) %>!% Coloniasaregenerallyunincorporatedandlowincomeresidentialsubdivisions,lackingbasic ' #' infrastructureand_services,_ԀalongtheborderbetweentheU.S.andMexico.Thesesettlements (#( havebeeninexistencefordecades,buttheexodusofthepoortocoloniasbeganinfullforce )$) duringthe1980sand1990s.Thelowcostoflandincoloniasprovidedopportunitiesforhome l*%* ownershipandrelieffromhigherhousingcostsinbordercities. R+&+  8,', _TheaimofthisreportwastoprovidequalitativeinformationonhowCensus2000was  conductedinselectedcolonias.Thisinformationcomesfromtwosources.Thefirstsourceis  fourstudiesconductedbyethnographerswithfieldworkexperienceincoloniasandwith | knowledgeofthesesettlementsworkingundercontractfortheCensusBureau.Theseconddata b sourceincludestheresultsoffocusgroupswithcensusenumeratorsandcrewleaderswho H workedintheselectedcoloniasstudiedbytheethnographers. . ~ Ethnographersfromfourcoloniasidentifiedanddocumentedthepresenceoffourmajorbarriers  J tocensusenumeration.Theseare:irregularhousing,littleornoknowledgeofEnglishand  0 limitedformaleducation,concernsregardingconfidentiality,andcomplexandfluidhouseholds.    However,theextenttowhichthesebarriersposedproblemsforCensus2000enumerationand    theCensusBureau'ssuccessindealingwiththeseobstaclesvariedacrossthefourcolonias.   " i "   iA623  0    Irregularhousingappearedtobeanobstacleinallfourcolonias.However,ethnographic ^  observationsrevealedthat,forthemostpart,censusenumeratorswereableto D  successfullynegotiatetheobstaclespresentedbyirregularhousing. iA6n6݌*z (#(# Ќ  " i "   i 823  0    LimitedreadingskillsandlittleornoknowledgeofEnglishwascitedasanobstacleto F enumerationinallfourcolonias.Regardlessofsite,theneedforaSpanishlanguage , censusformthatcanbeeasilyadministeredbyenumeratorsandreadilyunderstoodby  respondentswasdocumentedbyallethnographers. i 8:8݌(#(# Ќ  " i "   i6:23  0    Allethnographersreportedthatcoloniaresidentsexpressedconcernsregardingthe t confidentialityofcensusdata.Lackoftrustingovernmentandleerinessofnoncolonia Z residentsprevailedacrossallfoursites.However,itappearsthat,forthemostpart,these @ concernswerecounterbalancedbyCensusBureaueffortstopromoteCensus2000via &v paidadvertisementintheSpanishlanguagemedia. i6:c:݌ \(#(# Ќ  " i "   i<23  0    Complexhouseholdsandhouseholdswithmobileandambiguousmemberswere ( prevalentinallfourcolonias.However,thissituationwasparticularlypronouncedinthe  coloniasituatedinRiversideCounty,Californiabecauseofthesizeablenumberof  migrantworkersresidinginthiscounty. i<<݌ (#(# Ќ  AlthoughcoloniasontheU.S./Mexicoborderare,forthemostpart,ethnicallyhomogeneous, V"! thereisconsensusamongethnographersthatitisinappropriatetoassumethesamedegreeof <#" homogeneityonotherkeydimensionssuchaslanguage,theextentofsocialcohesion(i.e., "$r# community)amongcoloniaresidents,andthelevelofinfrastructuredevelopment. %X $ Recommendationsbasedonthisresearchinclude: &$"& " i "   i@23  0    Considerrevisingthetrainingmethodandtrainingmaterialsforenumeratorsandcrew (#( leadersworkingincolonias.Emphasizeclassroomtraininglessandemphasizeonthe )$) jobtraining. i@ A݌l*%*(#(# Ќ  _  8,', " i"   iB23  0    UseculturalfacilitatorsandpromotorasdevelopedbytheDallasROandimplementedin  theElPasoCounty,Texassiteasastartingpointandinitiateresearchthatwillinformthe  CensusBureauonhowtobestusetheseinitiativesincoloniasinallfourborderstates. iBB݌|(#(# Ќ  ! !" i"   iD23  0    ContinueandexpandthepracticeofhiringSpanishspeakingenumeratorswhoare H familiarwithcolonias. iDD݌. ~(#(# Ќ  @*" i"   iF23  0    ContinuetousetargetedpaidadvertisinginbothEnglishandSpanish. iF0F݌ J(#(# Ќ  ! !" i"   i"G23  0    InitiateresearchthatwillhelptheCensusBureaudetermineifmailingoutSpanish    languageformsin2010isaviablestrategy. i"GOG݌  (#(# Ќ  /OaCategoryKDataCapture  ^  SummariesofthetwoevaluationsintheDataCapturecategoryfollow. *z   EvaluationK.1.a,DCS2000DataCaptureAuditResolutionProcess (Rosenthal2003b )  F ThisevaluationlookedatresultsfromtheDataCaptureAuditResolution(DCAR)process.  DCARidentifiederrorsininterpretingscannedquestionnaires.  TheDCARconsistedofthreephases:anautomatedreviewofdatausedtosetpersonpanel0 '  3      ׀and t rosterentry1v '  4      ׀statuses,anedittocomparerespondentorenumeratorresponsesonhouseholdsize Z toahouseholdpopulationcountderivedfromatallyofpersonpanelsandrosterentries,anda @ clericalreviewofimagesandanupdateofdataforquestionnaireswhoseresponserecordshad &v conflictinghouseholdsizeinformation.  \ Thereweretwotypesofclericalreviewofcomputerimagesofquestionnairesthatfailededit: ( theAuditCountCheck(CountCheck)andtheAuditStatusReview(StatusReview).TheCount  CheckrequiredthatclerksreviewandcorrecttheOpticalMarkRecognition(OMR)  interpretationofrespondentorenumeratorresponsesonhouseholdsizeonly.Theydidnotmake   correctionstotheOMRfieldsbasedonareviewofpersonpanelsorrosterentries.TheStatus p!  ReviewrequiredthatclerksreviewandcorrecttheOMRfields.Inaddition,theyreviewedand V"! correctedthestatusofpersonpanelsandrosterentries. <#" TheCensus2000DataCaptureSystem(DCS2000)successfullycapturedtheresponsedatathat %X $ wereinputtothedeterminationofhouseholdsize.Itsuccessfullycapturednumericresponses %>!% andaccuratelyidentifiedthepresenceofresponsesincheckboxes. &$"&  ' #' Ofthe126,866,759returnsthatweresenttoDCAR,124,194,637returns(97.89percent)passed  theedit.Ofthe2,672,122failededits,theCountCheckprocessincluded882,555returns(33.03  percent)andtheStatusReviewprocessincluded1,789,567returns(66.97percent). |   H _TherateofeditfailuresvariedonlyslightlyacrossDataCaptureCenterwithinformtype.The  rateatwhichmailreturnspassedtheDCAReditvariedgreatlybyhouseholdsize.About98  percentofmailreturnswithahouseholdsizebetween1and9passedtheDCAReditandabout | 61percentofthemailreturnswithahouseholdsizeof10ormorepassedtheedit.Thelower b rateforlargehouseholdsmaybedueinparttothelimitof12namesthatcouldbereportedona H mailreturn.TherateatwhichenumeratorreturnspassedtheDCAReditvariedslightlyby . ~ householdsize.Theratedecreasedslightlyashouseholdsizeincreased.Therateforhouseholds  d withtenormorepersonsismuchlargerforenumeratorreturnscomparedtomailreturns,96  J percentversus61percent.  0 AsthecheckindateofthereturnbecamefurtherremovedfromCensusDay,thepercentsentto    CountCheckandStatusReviewincreasedformailreturnsfasterthanforenumeratorreturns,   indicatingmoreconsistentqualityforenumeratorreturnsovertime. x  Thestatusofpreauditduplicatesamongpersonpanelsandamongrosterentriesonmailreturns D  werecomparedtotheirpostStatusReviewstatus.Therewere52,406preauditduplicateperson *z  panelsand41,562preauditduplicaterosterentries.Only507(0.97percent)oftheperson ` panelsweredeterminednottobeaduplicateandonly1,233(2.97percent)oftherosterentries F weredeterminedtonotbeaduplicatebytheStatusReviewprocess.Thelowerrateofchange , forpersonpanelsmayindicatethatwithoutassociateddemographiccharacteristicsitismore  difficulttoaccuratelyidentifyduplicates.  TheStatusReviewchangedonlyasmallpercentageofpreauditstatuses.TheStatusReview t processchangedabout12percentofthestatusesforpersonpanelswithapre-auditstatusof Z valid.TheStatusReviewprocesschangedabout13percentofthestatusesforpersonpanels @ withapre-auditstatusofinvalid.TheStatusReviewprocesschangedabout29percentofthe &v statusesforshortformmailreturnrosterentrieswithapre-auditstatusofvalid.TheStatus  \ Reviewprocesschangedlessthan0.5percentofthestatusesforshortformmailreturnroster B entrieswithapre-auditstatusofinvalid.TheStatusReviewprocesschangedabout10percent ( ofthestatusesforlongformmailreturnrosterentrieswithapre-auditstatusofvalid.TheStatus  Reviewprocesschangedabout4percentofthestatusesforlongformmailreturnrosterentries  withapre-auditstatusofinvalid.   TheDCS2000successfullyinterpretedaveryhighpercentageofthehandwrittennumeric V"! entriesforhouseholdsize.Althoughtheoccurrenceofinterpretationerrorswaslowforboththe <#" mailandenumeratorreturns,theentriesmadebyenumeratorsweremisinterpretedabouttwice "$r# asoftenasthosemadebytherespondentstomailreturns.TheDCARprocesscorrectedthe %X $ householdsizeforabout64percentoftheenumeratorreturnsthatfailedtheCountCheckbut %>!% correctionweremadetoonlyabout32percentofthemailreturnsthatfailedtheCountCheck. &$"& TheDCARcorrectedthedataonalargenumberofcasesthatwouldhavebeenincludedinthe (#( CEFUhadtheDCARprocessnotbeenperformed. )$) Therecommendationistoincorporateasimilarprocessintothe2010Census. R+&+ _ 8,',  EvaluationK.1.b,EvaluationoftheQualityoftheDataCaptureSystemandtheImpactof  theDataCaptureModeontheDataQuality (Conklin2003)  Thepurposeofthisevaluationwastoseehowwellthereadingofcensusformscouldbe b delegatedtoautomateddatacaptureandimagingtechnology.Therawdataforthisevaluation H consistedofasampleof768,000shortformsand768,000longformsdistributedamong: . ~ _mailout_/mailbackshortandlongforms,enumeratorshortandlongforms,andU/Lshortand  d longforms.TheenumeratorandU/LformsincludedPuertoRicoandU.S.versions.The  J _mailout_/mailbackformsincludedbothEnglishandSpanishversions.  0 Productiontechnologyinvolvedthreemodes:OMRtodetermineinanautomatedprocessifa    checkboxontheformdidordidnothaveamarkinit;OMRtocapturethevalueofeach   characterinawriteinresponse;andKeyFromImage(KFI)modetohaveapersoninterpretand x  keyresponsesfromacomputerimageoftheform. ^  BoththeproductionautomatedtechnologyandtheevaluationKFIdatacaptureweresubjectto *z  anyoneofthefollowingerrors:failuretoreadafieldontheform,pickingupcontentthatwas ` notreallythere(asintryingtointerpretastraymark),incorrectlycapturingthecontentonthe F paper,orcorrectlycapturingwhattherespondentwrotebutnotwhattherespondentintended. , Thecheckboxcontentsmustbeidenticaltobeconsideredamatch.Pickingupthewrong  checkboxisahardmatcherror.Thewriteincontentsdidnothavetobeidenticaltobe  consideredamatch.Missingcharactersordroppingoraddingcharacterscanleadtosoftmatch t errors.Thedivergencebetweenthecontentswasscoredusingasoftmatchalgorithm.Asoft Z matcherroroccurredwhenthedivergencescoreexceededathreshold. @ Contentcandifferfromintent.Thiscanhappenforreasonssuchasstraymarksbeingreadas  \ charactersoriftherespondentwritespoorly.ThestandardforKeyFromPaper(KFP)wasto B capturecontentwithnomorethana2percente_rror_Ԁrate.Theperformanceoftheautomated ( technologydependedonwhetherthecharacterrecognitionalgorithmdeterminedthecontentwas  clearenoughtoprocess.Iftheautomatedtechnologydeterminedthecontentofawriteinfield  wasclear,itprocesseditwithatypicalerrorrateof1.0to1.1percent.Iftheautomated   technologydeterminedthecontentofacheckboxfieldwasclear,itprocesseditwithatypical p!  errorrateof1.2to1.5percent.Iftheautomatedtechnologyrejectedcontentasunclear,the V"! typicalerrorrateafterremedialkeyingbyhumanoperatorswas4.8to5.3percent. <#" Theevaluationconcludedwith90percentconfidencethemodesareallsignificantlydifferent %X $ fromoneanother.B  dThisstatementreflectsgeneralknowledgeofhow E)$( confidenceintervalsworkandisnotconsideredwithinthescopeoffactsrequiringindexingand 1*%) factchecking.dC  ԀTheKFImodetendstodealwithcontentparticularlyhardforhumanor %>!% machinetointerpretandthereforetheerrorrateisnotnecessarilyapoorreflectiononthe &$"& automatedtechnologyorthe_keyers._ԀTheerrorratesshowninthisevaluationshouldbe ' #' consideredconservativeupperlimitsforthetrueratesattributablesolelytothehardwareand (#( softwareconfigurationoftheautomatedtechnology. )$)   R+&+ Respondentreturnedformshadstatisticallysignificantlyhighernonblankhardorsoftmatch  errorratesforethnic,name,andracefieldscomparedtoenumeratorreturnedforms.Although  enumeratorreturnedformshadlowersoftmatcherrorratesfornamerelatedfieldscomparedto | respondentreturnedforms,theratesfornamerelatedfieldswerehighercomparedtoratesfor b otherfieldsonformsreturnedbyenumerators. H Forfieldsfilledoutforonlyonepersononaform,thehardorsoftmatcherrorratewas  d significantlyaffecteddependingonthespecificfieldbeingconsidered;formtypeorfield  J categorydidnothaveasignificantinfluence.Forfieldsfilledoutformultiplepersonsona  0 form,thesoftmatcherrorratewassignificantlyaffectedbyformtypeandfieldcategory.    Census2000RCChadasignificantinfluenceonthehardorsoftmatcherrorrate.Thesoft   matcherrorratefornamerelatedfieldsinRCCscoveringareasoftraditionalimmigrant x  concentrationinFlorida,LosAngeles,andNewYorkCitywassignificantlyhighercomparedto ^  otherRCCs. D  Forfieldsthatwerefilledoutforonlyonepersononaform,thelargestsignificantfactor ` affectingthenonblankerrorratewasform.Forfieldsthatwerefilledoutformultiplepersons F onaform,thelargestsignificantfactoraffectingthenonblankerrorratewasfieldcategory. , Theraceresponsehadastatisticallysignificanteffectonthenonblankerrorrate.Withinthe  limiteddatasetforrace,evaluatorswerenotabletofindindividualerrorratesthatwereoutliers.  Theeffectofracemaybepartofothersignificantfactors.Itwouldbehelpfultoincludeother t factorswithraceinafutureevaluation. Z Forfieldsfilledoutforonlyonepersononaform,theerrorratewasnotsignificantlyaffectedby &v datacapturemode.Forfieldsfilledoutformultiplepersonsonaform,thespecificfieldbeing  \ consideredandthedatacapturemodeinteractedtosignificantlyaffecttheerrorrate. B Themostfrequentcausesforfailingtocapturetheintendedresponsewereanextracheckbox,  missingcharacters,orawrongcharacter.Themostcommonreasonsfoundfortheseproblems  werepoorhandwriting,noreasonfound,orrulesnotfollowed.   Thefutureroleoftheautomatedtechnologyreducestotwopossibilities.Theautomated V"! technologyhasasupportingroleindecennialcensusprocessingortheautomatedtechnologyhas <#" adominantroleindecennialcensusprocessing. "$r# Severalpossibleresearchquestionsexistfortestsleadinguptothe2010Census. %>!% " i"   i23  0    ShouldtheCensusBureauexpandeffortstomakecertaingroupsoffieldseasierfor ' #' respondentstounderstandandfillout? i$݌(#((#(# Ќ  " i"   i=23  0    Dotheoutliererrorratesfornamerelatedfieldssuggestchallengestotheautomated l*%* technologythatrequireincreasedattention? i=j݌R+&+(#(# Ќ   8,', _" i"   i23  0    IstheespeciallyhighnonblankerrorratefornamerelatedfieldsinCensus2000RCCsof  traditionalimmigrantconcentrationsomethingthatrequiresmoreinvestigation? i܂݌(#(# Ќ  " i"   i#23  0    ShouldcertainfieldssentautomaticallytoKFIbeallowedtogothroughtheautomated b technologyforprocessing? i#P݌H(#(# Ќ  CategoryLProcessingSystems   J SummariesofthefiveevaluationsintheProcessingSystemscategoryfollow.     EvaluationL.2,OperationalAnalysisoftheDecennialResponseFileLinkingandSettingof   HousingUnitStatusandExpectedHouseholdSize (Rosenthal2003a) x  TheDecennialResponseFileisthefirstinaseriesoffileswhichultimatelyproducesfinal D  censuspopulationcounts.Areturnrepresentedasinglehouseholdenumeration.Areturn *z  consistedofoneormoreDecennialResponseFilehouseholdforms.Forexample,an ` enumeratorcontinuationformwaslinkedtoitsparentenumeratorformtocreateonereturn. F Similarly,aBeCountedFormforapartialhouseholdwaslinkedtoamailreturnformiftheBe , CountedFormcontainedinformationonadditionalhouseholdmembersnotincludedonthemail  returnquestionnaire.  Inmailbackareas,theuseoftwoformstoenumeratelargehouseholdsmostcommonlyoccurred t intheNRFUandCIFUoperations.Typicallythetwoformswereanenumeratorfirstformand Z anenumeratorcontinuationform. @ Largehouseholdsonmailreturnsgenerallydidnotinvolvelinking.TheCensusBureau  \ producedacompositerecordfortheselargehouseholdsfromtheoriginatingmailreturnanda B CEFUtelephoneinterview. ( TheCensusBureauassignedtoeachreturnahousingunitstatusand,ifthestatuswas  determinedtobeoccupied,anexpectedhouseholdsize.Then,theCensusBureauappliedthe   PrimarySelectionAlgorithm,acomputerprogramrunontheDecennialResponseFiletoselect p!  onereturntorepresenteachhousingunitinthecensus,ifmultiplereturnswerepresentfora V"! housingunit. <#" Thisevaluationpresentedtheresultsfromthelinkingofcensusformsandthesettingofhousing %X $ unitstatusandexpectedhouseholdsize. %>!% Of129,389,529returns,1,387,085returns(1.07percent)werelinked;thatis,theywerereturns ' #' comprisedoftwoormoreforms.Ofthese,39,108returns(2.82percent),hadthreeormore (#( forms. )$)   R+&+ TheTEAwiththehighestrateoflinkedreturnswasinU/E.Largehouseholdsprobablycaused  thisresult.Mostlinkedreturns(1,384,233returnsor99.79percent)werecomprisedofan  enumeratorfirstformandanenumeratorcontinuationform.Enumeratorsusedthiscombination | offormstoenumeratelargehouseholdsintheL/E,U/E,NRFU,andCIFUoperations.Linkage b ratescomparabletoU/EdidnotoccurinL/E,probablybecauseofaprocessingerror. H EnumeratorcontinuationformsinL/E"alongwithU/LaddsandU/Eadds"wereerroneously . ~ omittedfromtheDecennialResponseFile.  d Of129,389,529returns,1,318,350returns(1.02percent)hadoneormoreunresolvedcategories:  0 OccupiedwithUnresolvedPopulationCount,UnresolvedOccupied/Vacant,andUnresolved    Occupied/Vacant/Delete.    TheCensusBureausentmailreturnstoCEFUiftherewasaninconsistencyinhouseholdsize, x  leavingfewselfresponsereturnsunresolved. ^  AprogrammingerroraffectedthestatusresolutionforsomeVacantenumeratorreturns. *z  Mistakenly,anyInterviewSummaryPopulationof0wasrecodedtoblank.Asaresult,the ` CensusBureaumayhaveclassifiedupto133,438VacantreturnsasDeletesandupto258,963 F VacantreturnsasUnresolvedOccupied/Vacant.Asmanyas145,367housingunitsofthe , 191,826housingunitsinthecensusthathadtheiroccupancystatusimputed(75.78percent)may  havebeenaffectedbythislattererror.  Therewere712,858unresolvedenumeratorreturns(1.51percent),329,895returnswere t OccupiedwithUnresolvedPopulationCount,329,266returnswereUnresolved Z Occupied/Vacant,and53,697returnswereUnresolvedOccupied/Vacant/Delete. @ Ѐ &v Foroccupiedselfresponse(restrictedtopapermailbackquestionnaires)andenumeratorreturns,  \ settingtheexpectedhouseholdsizewasusuallystraightforward.For74,725,437selfresponse B returns(93.71percent)thenumberofvalidpersonrecordsandrosternamescorrespondedtothe ( respondentreportedhouseholdsize.Forenumeratorreturns,mosthouseholdsizemeasuresalso  wereconsistent.Forlinkedreturnsorunlinkedreturnsthathadthe continuationform(s)  attachedboxcheckedandtheexpectedhouseholdsizeequaltotheInterviewSummary   Population,1,475,382returns(99.11percent)hadthesameInterviewSummaryPopulationand p!  respondentreportedhouseholdsize.ForunlinkedreturnswiththeInterviewSummary V"! Populationlessthanorequaltofive,26,897,133returns(99.52percent)hadthesameInterview <#" SummaryPopulation,numberofvalidpersonrecords,andrespondentreportedhouseholdsize. "$r# ForunlinkedreturnswiththeInterviewSummaryPopulationgreaterthanfiveandthe %X $ respondentreportedhouseholdsizegreaterthanfive,68,599returns(99.58percent)hadthe %>!% sameInterviewSummaryPopulationandrespondentreportedhouseholdsize. &$"& Recommendationsinclude: (#(  * #$%&'((xx* #$%&'(" i"   i23  0    Attempttolinkonlyenumeratorfirstandenumeratorcontinuationforms,ifsuchforms l*%* existinthefuture.Doingsowouldsimplifythelinkingprocess,causeverylittlelossof R+&+ data,andwouldhavealmostnoeffectonthepopulationcounts. i1݌8,',(#(# Ќ  _" i"   i23  0    RedesigntheInterviewSummarySection,ifthissectionexistsinthefuture,toimprove  theconsistencyofresponses. i݌(#(# Ќ  " i"   i)23  0    Usehandheldcomputerstohelpensuredatacaptureandconsistencyofresponses. i)V݌b(#(# Ќ  EvaluationL.3.a,AnalysisofthePrimarySelectionAlgorithm (Baumgardner2002) . ~ Thefullreportforthisevaluationisnotavailablebecauseitcontainsproprietaryinformation.  J Whilevariousmethodsofcollectingdatawereimplementedwiththedesireofobtainingamore    accuratecensuscount,thevariousmethodsalsopresentedthepossibilityofreceivingmultiple    responsesforasingleCensusID.ThePrimarySelectionAlgorithmwasthecomputerprogram   designedtoresolvethereceiptofmultipleresponsesfromhousingunits.Majorfeaturesofthe x  Census2000PrimarySelectionAlgorithmdesignincludedperformingpersonmatchingbetween ^  returns,constructingPrimarySelectionAlgorithmhouseholds,selectingtheprimaryPrimary D  SelectionAlgorithmhousehold,andselectingadditionalpersonsforthecensushouseholdthat *z  werenotintheprimaryPrimarySelectionAlgorithmhousehold. ` Lessthan10percentofallCensusIDsontheDecennialResponseFilewereenumeratedby , morethanonereturn.Morethan95percentofthesewereenumeratedbyonlytworeturns.  About55percentoftheCensusIDsenumeratedbytworeturnsweretheresultoftwo  enumeratorreturnsandabout82percentoftheseweretheresultofreturnsfromNRFUand  CIFU.AboutathirdofallCensusIDswithtworeturnsconsistedofoneenumeratorandone t mailreturn;about96percentoftheseweretheresultofamailbackreturnandaNRFUreturn. Z SomereturnsweredefinedasineligibleforthePrimarySelectionAlgorithmprocess.There &v were2,656,951ineligiblereturnsatallCensusIDs.Morethan67percentofthesereturnswere  \ ineligibleduetobeingclassifiedasadeletedhousingunitrecord.Takingtheseineligiblereturns B outoftheuniverse,8,960,245CensusIDs(lessthan8percentofCensusIDsontheDecennial ( ResponseFile)hadmorethanoneeligiblereturn.  APrimarySelectionAlgorithmhouseholdwasasetofassociatedpersonsatoneCensusID.   Thesetmaycontainnopersons(avacantPrimarySelectionAlgorithmhousehold)oroneor p!  morepersons.IftwoormorereturnsforthesameCensusIDhadatleastonepersonincommon V"! (determinedbypersonmatching),thenthesereturnsformedasinglePrimarySelection <#" Algorithmhousehold. "$r# Over73percentofCensusIDswithmultipleeligiblereturnshadjustonePrimarySelection %>!% Algorithmhousehold.CensusIDswithtwoormorePrimarySelectionAlgorithmhouseholds &$"& accountedforjustover2percentofallDecennialResponseFileCensusIDs.Theprimary ' #' PrimarySelectionAlgorithmhouseholdwasthePrimarySelectionAlgorithmhouseholdthat (#( wasusedinfurtherprocessing.WhenmorethanonePrimarySelectionAlgorithmhousehold )$) existed,theprimaryPrimarySelectionAlgorithmhouseholdwasselectedbysequentially l*%* applyingcriteriatoallofthePrimarySelectionAlgorithmhouseholdsuntilonlyonePrimary R+&+ SelectionAlgorithmhouseholdwasselected. 8,', ЇMostPrimarySelectionAlgorithmhouseholdsatCensusIDswithmultiplereturnsconsistedof  oneortworeturns.TworeturnPrimarySelectionAlgorithmhouseholdsweremostoften  formedbytwoenumeratorreturnsoronemailreturncombinedwithoneenumeratorreturn. | WhentwoenumeratorreturnsformedaPrimarySelectionAlgorithmhousehold,over91percent b weretheresultofonereturnfromNRFUandonereturnfromCIFU.Thiswasexpecteddueto H thedesignoftheCIFUoperation. . ~ Ofthe8,716,359CensusIDswithtwoeligiblereturns,over70percenthadaredundantreturn(a  J returncontainingonlypersonrecordsrepresentedonthebasicreturnofaPrimarySelection  0 Algorithmhousehold)andalmost57percentoftheseredundantreturnswerenotvacant.    Almost85percentofallredundantreturnswereenumeratorreturns.Morethan55percentof   redundantenumeratorreturnsresultedfromNRFUandnearly88percentofthesewereoccupied, x  mostlikelyduetothereceiptofalatemailreturn.About43percentofredundantenumerator ^  returnsresultedfromCIFUand97percentofthesewerevacant,asexpected. D  Ofthe2,349,988CensusIDswithtwoPrimarySelectionAlgorithmhouseholds,morethanhalf ` hadanenumeratorreturnasthebasicreturnofbothPrimarySelectionAlgorithmhouseholds. F Nearly80percentofthesecasesresultedfromonereturnfromNRFUandonereturnfrom , CIFU.ThisismostlikelyduetoavacantreturnfromtheNRFUoperationandanoccupied  returnfromtheCIFUoperation.  MostcasesofmultipleenumerationsinCensus2000wereexpectedasaresultofthedesignof t censusoperations.ThePrimarySelectionAlgorithmwasdesignedtoberobustandhandleas Z manyunusualcasesaspossible,buttheresultsshowthattheseunusualcaseswereveryfew.If @ thisoperationisimplementedinthesamemannerinthefuture,aprocessingstepdonepriorto &v thePrimarySelectionAlgorithmshouldremovefromfurtherprocessingaCIFU_return_Ԁthatjust  \ confirmsthestatusofaNRFUreturn.TheNRFUreturninthiscaseshouldbeflaggedto B indicatethatitsstatuswasconfirmed. (  EvaluationL.3.b,ResolutionofMultipleCensusReturnsUsingaReinterview   (Baumgardner2003)   Thefullreportforthisevaluationisnotavailablebecauseitcontainsproprietaryinformation. V"! ThefocusofthisevaluationwastoexaminewhetherornotthePrimarySelectionAlgorithm "$r# madethebestdecisionsonthedeterminationofthecensushousehold,giventheinformation %X $ collectedfromreturnsfortheCensusID.TheanalysisconcentratedonCensusIDswithtwo %>!% returnssince97percentofallCensusIDswithmultiplereturnshadexactlytworeturns. &$"& AsampleofCensusIDsaffectedbythePrimarySelectionAlgorithmwasselected.An (#( interviewwasconductedateachCensusIDwithsomeonefamiliarwiththehousehold )$) enumeratedduringCensus2000.Thegoalofthereinterviewwastodeterminetheresidency l*%* statusofeachpersononthecensusreturnsattheCensusID.Theresidencystatusesobtainedin  R+&+ thereinterviewwerethenusedtodetermineifthePrimarySelectionAlgorithmmadethe best  decisionsonthedeterminationofthecensushousehold.  _ XXz#XzX _ #Onesetoflimitationstothisstudyincludesoperationalproblemswithdatacollection.Unit u nonresponse,allowingtheuseofproxyrespondents,andrecallbiasmaycontributetomissingor [ inaccurateresponses.Anothersetoflimitationstothisstudyincludeslimitstothestatistical A  analysis.Thisstudydidnotinvestigatevacantreturnsorevaluatethepersonmatchingprocess. ' w Therewerealsopersonrecordswithanunresolvedresidencystatusanderrorsintheinclusionof  ] someCensusIDsinthesample.Thesefactorsmakeitdifficulttofullyevaluatetheperformance  C ofthePrimarySelectionAlgorithm.  )  WhenpersonmatchingdidnotmatchpeopleacrossthetworeturnsataCensusID,thesetwo   returnsformedtwoseparatePrimarySelectionAlgorithmhouseholds.AtCensusIDswithtwo   PrimarySelectionAlgorithmhouseholds,thereinterviewdeterminedthattherewereresidentsin q  bothhouseholdsabout38percentofthetime,residentsinjustoneofthehouseholdsabout58 W  percentofthetime,andnoresidentsineitherhouseholdabout4percentofthetime. =  AtCensusIDswithtwoPrimarySelectionAlgorithmhouseholdswherethereinterview  Y determinedthattherewereresidentsinbothofthehouseholds,personmatchingwasperformed ? andmissedaduplicateidentifiedduringthereinterviewinabout16percentofthecases.Also % ofinterestishowoftenthePrimarySelectionAlgorithmpickedthe bestPrimarySelection   AlgorithmhouseholdtorepresenttheCensusIDsincebothofthehouseholdscontained  residents. BestisdefinedhereasthePrimarySelectionAlgorithmhouseholdwiththegreater  netnumberofresidents.Thenetnumberofresidentsisthebalanceaftersubtractingthenumber m ofnonresidentsfromthenumberofresidents.AtCensusIDswithresidentsintwoPrimary S SelectionAlgorithmhouseholds,the besthouseholdorahouseholdwhichwasidenticalin 9 termsofnetresidentstotheotherhouseholdattheCensusIDwasselectedabout80percentof o thetime. U Nearly58percentofCensusIDswithtwoPrimarySelectionAlgorithmhouseholdshadatleast ! oneresidentinjustoneofthosehouseholds.Atabout65percentoftheseCensusIDs,the  PrimarySelectionAlgorithmselectedthehouseholdthatcontainedatleastoneresident.Ofthe   CensusIDswithtwoPrimarySelectionAlgorithmhouseholdsandresidentsinjustoneofthose !  households,theeffectivenessofthePrimarySelectionAlgorithmhouseholdselectioncriteria i"! wasexamined.Asexpected,higherpriorityselectioncriteriaweremoreeffectiveatselecting O#" thecorrectPrimarySelectionAlgorithmhouseholdthanthelowerpriorityselectioncriteria. 5$# However,theselectioncriterion CEFUStatusdidnotperformaswellasexpected.Itselected %k $ thecorrectPrimarySelectionAlgorithmhouseholdonlyabout69percentofthetimethatitwas &Q!% used. &7"& Inthefuture,anevaluationsuchasthisoneshouldsetouttodetermineifthePrimarySelection ($( AlgorithmselectedtherighthouseholdregardlessofwhetherornotthePrimarySelection )$) Algorithmhouseholdisoccupied.Furthermore,thequestionnaireusedforthereinterview *%* shouldbedesignedtomanagebothoccupiedandvacantreturns.Itshouldalsonotrelyon e+&+ censusresidencerulestodetermineresidencystatusforpeopleattheCensusID,sincethe K,', PrimarySelectionAlgorithmitselfcannottakethoserulesintoaccountwhenmakingdecisions  regardingthecensushousehold.   EvaluationL.4,CensusUneditedFileCreation (Jonas2003a) b TheCensus2000HundredPercentCensusUneditedFilecontainsallthehouseholdandperson . ~ recordsincludedinCensus2000.IthasalltheattributesofthefinalCensusfile,exceptthe  d imputationofpersoncharacteristicswhereneeded.ThepurposeofCensusUneditedFile  J creationistodeterminewhichaddressesareinthecensusandtodeterminethecountofpersons  0 ateachsuchaddress.    Nearly128millionaddresseswereeitherontheDMAFasCensus2000beganorwereaddedto   itinthecourseofCensus2000operations.Priortounduplicationprocedures,_approximately_ x  117.3millionwereultimatelyresolvedashousingunitaddresses.Justover9millionaddresses ^  weredeterminedtonotbevalidaddressesandroughly1.4millionaddressesweredeterminedto D  benonvalidduplicatesofvalidaddressesontheDMAF. *z  Ofthe117.3millionaddressesresolvedashousingunitaddresses,106.7millionwere F determinedorimputedtobeoccupiedandtheremaining10.6millionweredeterminedor , imputedtobevacant.  Roughlyhalfamillionaddresseshadtheirstatusresolvedbyimputation.Therewere195,245  addressesdeterminedtobevalidCensusaddresseswhoseoccupancystatuscouldnotbe t determinedandhadtobeimputed.Therewere296,617addresseswhosevalidityasCensus Z addressescouldnotbedetermined.Asaresult,theirvalidityandtheiroccupancystatuswere @ bothimputed.TherewasnoenumerationdataontheDecennialResponseFileortheDMAFfor &v 251,477(84.8percent)oftheaddresseswhosevalidityasCensusaddressescouldnotbe  \ determined. B RecommendationsforsomechangesaffectingCensusUneditedFilecreationinclude:  " i"   i23  0    UsestrongersoftwareQAprocessestoensuremorecompleteadherenceto   specifications. i݌p! (#(# Ќ  " i"   i23  0    Refinethetimingoflatecensusfollowupoperationstoensurethataddressesaddedby <#" thoseoperationsareplacedontheDMAFintimeforthequestionnairesfromthose "$r# addressestobeincludedinthecensus. i݌%X $(#(# Ќ   &$"& _EvaluationL.5,OperationalRequirementsStudy:TheBetaSiteSystemsTestingand  ManagementFacility (TitanSystemsCorporation2003)  ThisstudyassessedtheextenttowhichtherequirementsfortheBetaSiteoperationandits b internalprocessessupportedvariousautomatedsystemsusedduringCensus2000.Thefindings H presentedarequalitativeinnatureastheyreflectthevariedopinionsandinsightsoftheBetaSite . ~ operationspersonnelandcustomerswhowereinterviewedbytheTitanSystemsCorporation.  d TheBetaSitewasasoftwareevaluationfacilitywithintheCensusBureauthatwasinvolvedin  0 thetestinganddeploymentofCensus2000systemsandrelatedcomponents.Itsprimary    objectivewastoassessasystemsdeploymentreadiness;however,italsoconductedsecurity    testing,providedsoftwarereleaseservices,andperformednetworkmonitoringand   troubleshootingsupport. x  SecurityevaluationwasadistinctphaseoftheBetaSitetesting.TheBetaSitepersonnelworked D  inacooperativefashionwiththeInformationTechnologySecurityOfficetoassureappropriate *z  securityconsiderationswereproactivelyaddressed.Overall,thestructureofthetesting ` processesandassociatedfunctionswerecomprehensiveandwerealignedtosupportthe F objectivesoftheBetaSite. , PlanningfortheBetaSitesupportforCensus2000beganinmid1996andcontinuedthroughthe  censustoaccommodatechangingoperationalrequirements,asneeded.Thephysicalsitewas  constructedin1996.InadditiontotestingCensus2000systems,theBetaSitehadtoaddress t suchchallengesasrampingupthetestinginfrastructureandperformingYear2000compliance Z testing.AccordingtoapostassessmentstudyoftheBetaSite,over1,200softwaretestswere @ performedbytheBetaSitefromlateFiscalYear1997throughFiscalYear2000.TheBetaSite &v alsomaintainedsystemconfigurationsforover8,000personalcomputersand570servers.  \ GiventhesuccessofCensus2000andtheunprecedentedrelianceonautomatedsystems,itis ( evidentthattheBetaSiteplayedanimportantroleinthedecennialcensusandcontributed  significantlytoitssuccess  AlthoughthesoftwarevalidationroleoftheBetaSiteoperationwaswidelyseenwithinthe p!  CensusBureauasbeinganecessaryfunction,manyoftheBetaSitescustomersexpressed V"! concernsovertheefficiency,consistency,andtimelinessofthetestingprocessesthatwere <#" employed.TherequirementsfortheBetaSitecouldhavefocusedmoreattentionontheimpact "$r# thatitsinternalprocesseswouldhaveoncustomersoperations.Conversely,developersneeded %X $ tofactorintimeforBetaSitetestingintheirdevelopmentprocess.Inthisregard,theBetaSite %>!% personnelnotedthat,fromtheirperspective,therewerealargenumberofurgentrequeststhat &$"& mighthavebeenavoidedwithproperplanning/schedulingbytheprogramoffices. ' #' Theissueofwhowasresponsiblefordevelopingtestplanswasnotfullyaddressed.Although )$) theProgramMasterPlandiscussedtheBetaSiteWorkflowandthe receiptofrequirements, l*%* whichincludedtestplans,data,andcasesfromthedeveloper,theprecisionofthose R+&+ requirementswasneverfullydefined.TheperceptionoftheBetaSitestaffwasthat 8,', responsibilityfortestplanswasdefinedthroughmeetingswithcustomers.Indiscussionswith  testersandcustomersalike,itwasevidentthatthisissuewasnotfullyresolvedduringthe  requirementsphase. | Earlyplanningmayhaveaddressedrequirementsfortwowaycommunications.Althoughaset H ofphysical,logistical,andproceduralrequirementswasoutlinedinApril1997,theydidnot . ~ adequatelyaddresstheneedforastructuretoensureeffectivecommunicationsbetweentheBeta  d Sitetestersanddevelopers.InterviewsconfirmedthattheBetaSiteprocesswasoftenunclearto  J mostcustomersandthisledtoasignificantnumberofcommunicationdifficulties,especially  0 whentheneedarosetoescalateissueresolutiontoahigherlevel.    In1997,theBetaSitemanagementoptedtousetheservicesoftheGeneralService   AdministrationsFederalSystemsIntegrationandManagementCenterasameansofacquiringa x  capableprimecontractorfortheBetaSite.TheFederalSystemsIntegrationandManagement ^  Centermanagesmultipleawardcontractswithqualifiedsystemintegratorswhocanbe D  competitivelyselectedinarelativelyshortperiodoftime.Resourcespermitting,itmayhave *z  beenbeneficialfortheBetaSitetousetheFederalSystemsIntegrationandManagementCenter ` intherequirementsplanningarea. F RequirementsdidnotgiveadequateconsiderationtothecomplexitiesofmanagingtheCensus  2000systemsinanetworkedenvironmentandthedivisionofresponsibilitieswithintheCensus  Bureau'stechnicalmanagementinfrastructurewasproblematic.Thiswasacrucialissueinview  ofthesizeofthenetwork.Duringthedecennialcensus,networkadministrationand t configurationissuesarosebetweentheTechnologiesManagementOfficeandtheBetaSite. Z Theseandotherfindingsledtothefollowingrecommendations: &v " i"   i23  0    Fullyconsiderrequirementsforcommunicationprocesses. i݌B(#(# Ќ  " i"   i23  0    ݚImprovetestersknowledgeaboutthepurpose,use,andcapabilitiesofthesoftware. i݌(#(# Ќ  " i"   i23  0    Majoradvancementsintechnologywillrequireearlyscopingofthelevelofeffort   requiredfromtheBetaSitein2010. i݌p! (#(# Ќ  " i"   i23  0    ݚImprovelifecyclemodelforthe2010Census. i݌<#"(#(# Ќ  _  %>!% CategoryMQualityAssuranceEvaluations   SummariesofthetwoevaluationsintheQAEvaluationscategoryfollow. |  EvaluationM.1,EvaluationoftheCensus2000QAPhilosophyandApproachUsedinthe H AddressListDevelopmentandEnumerationOperations (Morgansteinetal2003) . ~ Census2000involvednumerousfieldoperationsand,atitspeak,employedalmostahalfmillion  J temporaryworkers.Managingthequalityofthedataproducedbythislarge,decentralized,and  0 transientworkforcewasamajorchallenge,whichtheCensusBureauattemptedtomeetby    introducinganextensiveQAprogramintoitsongoingoperations.TheobjectivesofthisQA    weretominimizesignificantperformanceerrors,topreventtheclusteringofsignificant   performanceerrors,andtopromotecontinuousimprovement. x  EvaluationM.1evaluatedtheeffectivenessoftheQAprogramsdevelopedforandimplemented D  inthemajorfieldoperationsinCensus2000and,specifically,thoseoperationsusedtoupdate *z  theCensusBureausaddresslistanddirectlyenumeratethepopulation.Further,itidentified ` strengthsandmajordeficienciesandprovidedacritiqueoftheCensusBureausQAphilosophy. F Theevaluation,whichwasconductedbyWestat,wasaccomplishedthroughacombinationof  approaches,includingareviewofmaterialspreparedforuseintrainingandincollectingdata,  examiningobservationreports,interviewingarangeofCensusBureaustaffwhohadbeen  closelyinvolvedinthemanyphasesoftheQAprogram,andbyexaminingsuchmaterialsas t wereavailableconcerningtheoperations,aswellasthesuccessesorfailuresoftheQA Z programs.Asafinalstep,Westathelddiscussionswithstaffinthestatisticalofficesofthe @ UnitedKingdom,Canada,andAustralia,allofwhichhadrecentlyconductedanationalcensus, &v todeterminehowthesecountriesapproachedtheapplicationofQAintheircensusprograms,  \ andthepossiblerelevanceoftheiractionstofutureQAplanningoftheCensusBureau. B Census2000continuedthetradition,in_itiated_Ԁinthe1960Census,ofincorporatingintofield  operationsnumerousactivitiesdescribedasQA.ThiscommitmenttoqualityandQAcertainly  isasignificant strength.TheCensusBureaualsometitsobjectiveforQAthatitbe   completelytransparentinCensus2000.Tothatend,materialsusedtotrainenumeratorsand p!  firstlevelsupervisorscontainedspecificreferencesastowhyQAwasimportantandastohowit V"! wouldbeimplementedandallenumeratorswereexposedtotheconceptofandneedfor <#"  qualityperformanceand,generally,weremeasuredagainsttheestablishedstandards._Finally,_ "$r# mostoperations,hadsomeformofQAprocessinplace. %X $ AreviewofobservationreportsandinterviewswithCensusBureaustaffwhowereinvolvedin &$"& theQAprogramrevealedthattheoverallperceptionwasthattheCensus2000QAfieldprogram ' #' wasanimportantelementinpreventingsignificanterrorsandinpreventingtheclusteringof (#( significanterrors.Althougherrorsofbothtypesdidoccur,forthemostpart,theywerecaught )$) expeditiouslyandrectified.Onthisbasis,theQAfieldprogramscanbeviewed,generally,as l*%* successfullymeetingthefirsttwoelementsoftheCensusBureausQAmissiontoprevent R+&+ significanterrorsandtopreventtheclusteringofsignificanterrors. 8,', ЇAstotheCensusBureausgoalof makingthebestuseoftheavailabletechnologyand  statisticalprocesstoolswithrespecttoitsdesiretopromotetimelyandcontinuousimprovement  throughoutthefieldoperations,theauthorsconcludedthattheanswerismixed.Basedonthe | CensusBureausevaluationstodateandcommentsfromthoseinvolved,manyoftheearly b activitiesinpreparingforCensus2000wereseenashavingutilizedafullQAapproachthatmet H theCensusBureausstatedgoalofpromotingtimelyandcontinuousimprovement.Examplesof . ~ activitiesconsideredashavingsuccessfullyutilizedaQAapproachincludepreparingthe  d geographicframeworkandprintingquestionnairesandrelatedforms.However,inthecontextof  J whatactuallytranspiredduringthedatacollectionphase,theperceptionwaslessclearand  0 decidedlymixed.    QAwasnotperceivedasanequalpartner,norwasQAstaffgiveneitherthenecessaryauthority   ortherequiredfreedomtocompleteitstasksuccessfully.Finally,dedicatedQAstaffwasnot x  assignedtoRCCs.AvitalaspectoftheQAprogramforpromotingcontinuousimprovement, ^  realtimecaptureanddisseminationofdataduringthedatacollectionprocess,withwhichto D  monitor,evaluate,andreact,wasnotimplemented. *z  Somekeysuggestionsarenotedbelow: F  * #$%&'((xx * #$%&'(" i"   iK23  0    EnsurethattheQAeffortisseenasanintegralandimportantelementinthe2010Census  program.Tothatend,provideadequateresources,bothinfundingandstaff,andinitiate  earlyplanning,research,andtestingfortheQAphase. iKx݌(#(# Ќ  " i"   i23  0    InvolvetheexecutivestaffinsupportingandmonitoringQAefforts,especially Z throughoutthedatacollectionphase. iD݌@(#(# Ќ  " i"   iW23  0    EstablishaseniormanagementteamtocoordinateandapprovetheoverallQAplanand,  \ throughoutthedecennialperiod,toreviewprogressandresolveissues. iW݌B(#(# Ќ  " i"   i23  0    EstablishtheequalityofQArelativetoproduction.Simplyput,QAmustbeseen,  understood,andacceptedasanessentialelementofthecensusandasanequalpartnerat  alllevelsofplan_ning_,implementation,andreview. i݌ (#(# Ќ  " i"   i23  0    DevelopandimplementaManagementInformationSystemcomponentwhichprovides V"! management,inrealtime,withrelevantinformationonthequalityofthedatacollection <#" elements. i݌"$r#(#(# Ќ  " i"   i/23  0    ExpeditethedocumentationofCensus2000andestablishreadyaccesstothe %>!% information.SomeexamplesincludetheQAevaluationprogramandthedocumentation &$"& oftheexperiences,problems,andsolutions,suggestions,andrecommendationsofstaff, ' #' andtheaccumulationofmemorandadetailingproblems,issues,andsolutions. i/\݌(#((#(# Ќ   )$)  EvaluationM.2,EffectivenessofVariablesUsedintheModeltoDetectDiscrepantResults  DuringReinterviewandtheIdentificationofNewVariables (Johanson2003)  TheCensus2000NRFUReinterviewprogramincludedthreecomponents:arandom b reinterview,anadministrativereinterview,andasupplementalreinterview.Aportionof H completedenumeratorquestionnaireswereselectedtobereinterviewedand,oncethe . ~ reinterviewwasconducted,theunitstatusandhouseholdrosterwerecomparedtotheoriginal  d enumeration.Thepurposeofthereinterviewprogramwastoidentifyfaultydatacollection,both  J intentionalandunintentional.  0 Therandomreinterviewcomponentwasdesignedtoverifyworkfromeachenumerator.    Virtuallyallenumeratorswhocompletedaminimumoftenenumeratorquestionnaireshadone   ormoreoftheirquestionnairesselectedforrandomreinterview.Randomreinterviews x  represented93.09percentofthecasesselectedforthereinterviewprogram.Theremainderof ^  thereinterviewcaseswereadministrativeandsupplementalreinterviewcases(4.34percentand D  2.57percent,respectively). *z  Outlierenumeratorswereidentifiedforadministrativereinterviewbycomparingquestionnaire F characteristicsofeachenumeratoragainsttheaveragefortheirarea.Ahighvacancyrate,ahigh , rateofpartialinterviews,ahighdeleterate,ahighrateofquestionnaireswithapopulationcount  ofone,anddifferencesinaveragepopulationperhouseholdwerevariablesusedinthe  comparison.Thereportsidentifyingtheseoutlierenumeratorswererunonceaweek.Overthe  entireNRFUoperation,291,441enumeratorswereflaggedasoutliersforoneofthereasons t above.Thisis62.57percentofenumeratorswithcompletedwork.Notalloftheseenumerators Z hadcasesselectedforadministrativereinterview.Atthediscretionofsupervisors, @ approximately5percentofenumeratorsflaggedforadministrativereinterviewhad &v administrativecasesselected,or3.5percentofallNRFUenumerators.  \ Supplementalreinterviewcouldbeusedanytimetherewasreasontosuspectcasesmightnotbe ( completedcorrectly.Supplementalcaseswithcompletereinterviewinformationshoweda  higherfrequencyofenumeratorerrorbetweentheoriginalenumerationandthereinterview  (11.30percent)thanrandomandadministrativecases(9.42percentand9.67percent,   respectively).Thishigherincidenceoferroridentificationshowstheeffectivenessofthe p!  supplementalreinterviewcomponent. V"! Attheindividualcaselevel,administrativeandrandomreinterviewfoundasimilarproportionof "$r# caseswithdiscrepancies.Oftheenumeratorsinadministrativereinterview,52.09percenthad %X $ oneormorecasesinerror.Thisismuchhigherthanthepercentofenumeratorsinrandom %>!% reinterviewwitherrorcases.Therangeoferrorratesforenumerators,dependingonthenumber &$"& ofrandomreinterviewcasesselected,wasapproximately10to14percent. ' #' Theevaluationincludedadiscriminantregressionmodeltoevaluatetheeffectivenessofvarious )$) variablesintheadministrativereinterview.Evaluatorswerelookingforinteractionsthatmight l*%* helpimprovethemodeltoidentifyoutlierenumeratorsfortheadministrativereinterview.Ofthe R+&+ characteristicsreviewedfortheadministrativesample,thehighdeletevariablehadthebiggest 8,', _impactforidentifyingenumeratorswitherror.However,theregressionmodelsshowedthat  verylittleofthevarianceassociatedwiththedependentvariable(presenceoferror)was  explainedbytheindependentvariablesofinterest.Thisindicatesthatthedependentvariable | couldbeexpectedtobehavesimilarlyforrandomlyselectedenumeratorsandenumerators b identifiedasoutliers. H Interpretingtheseresultswasdifficultbecauseofoperationallimitations.Theanalysisshowed  d thatadministrativereinterviewwasdefinitelyeffectiveinidentifyingenumeratorswithhigh  J errorrates,yetthecontributionofthevariablesusedtoselecttheenumeratorswasnot  0 meaningful.Thisispartiallyexplainedbythefactthatalthoughenumeratorsweretargeted    basedonworkcharacteristics,theselectionofcasesforreinterviewdidnotreflectthe    characteristic(s)thatcausedtheenumeratorsoutlierstatus.Otherstudiesandacademicexperts   haveshownthatatargetedreinterviewcanbeverybeneficial.Anexampleoftargeted x  reinterviewwouldbeifanenumeratorwasanoutlierforhighdeletes,thendeletedcaseswere ^  specificallyreinterviewed.Usingtargetedcaseselectionwilllikelymaketheadministrative D  reinterviewmoreeffective. *z  Recommendationsincludedevelopingprocedurestoreducethenumberofenumerators F identifiedasoutliers.Thiscanbeaccomplishedbyincreasingthecriticalcutofflevelsor , accountingformultipletestsandtherecurringtimeperiods.Reducingtheoutliersidentifiedby  theadministrativetestwillcreateasmallerworkloadtoreview.This,inturn,shouldincrease  thepercentageofenumeratorsreinterviewedandtheeffectivenessoftheadministrative  reinterviewprograminidentifyingenumeratorswithdiscrepantresults. t  CategoryQOrganization,Budget,andManagementInformationSystem  &v  EvaluationQ.1,ManagementEvaluationofCensus2000 (IBMBusinessConsultingServices B 2003) ( Thisevaluationdocumentedtheeffectivenessofthemanagementapproachusedinthe  DecennialCensusineachofsevenareasincludingthemanagementmodel,organizational   structuresandprocesses,thedecisionmakingprocess,managementinformationtools,staffing, p!  externalinfluences,andtheuseofcontracts. V"! Theevaluationwasconductedbyacontractor,IBMBusinessConsultingServices.The "$r# contractorsunderstandingoftheexternalandinternalenvironmentformedthefoundationupon %X $ whichthedetailedapproachforassessingorganizationalandmanagementperformancewas %>!% based.Thecontractorevaluatedthemanagementapproachbyconductinginterviewswithkey &$"& CensusBureaustaff.Atotalof52interviewswereconductedwithCensusBureaupersonnel ' #' includingnineExecutiveStaff,13DivisionChiefs(includingRegionalDirectors),18Assistant (#( DivisionChiefs,and12BranchChiefs.Thecontractorinterpretedtheinformationcollectedin )$) theinterviewsusingmanagementtheoryandexperiencefrompreviousmanagementreviewsto l*%* evaluatethedecennialcensusmanagementpractices.Theintervieweeshadvaryinglevelsof R+&+ exposuretotheissueareasmentionedintheInterviewGuide,andtheygenerallycommentedon 8,', areaswithwhichtheyhaddirectknowledgeorexperience.Therewasnoattempttostratifythe  commentsfromtheinterviewees,sinceallthecommentsweretreatedequally,independentof  organizationalunit,experience,orotherdifferences. | PerformanceMeasures: H Keyperformanceindicatorsrevealedthat,incertainrespects,Census2000wasthemost  d successfuldecennialcensuseverconducted.AccordingtoseniorCensusBureauofficials,the  J mostcriticalperformancemeasurewasNetCoverage.InCensus2000,thepercentofnet  0 undercountestimatewasminus0.49.Anetundercountestimateofminus0.49representsa    smallestimatedovercountofthehouseholdpopulationforCensus2000.Achievementofa    smallnetcoverageerrorthatisclosetozeroisanimportantsuccessfactor.   ThenationalresponseratethatdeterminedtheNRFUworkloadwas65percent,whichmatched ^  the65percentresponseratefromthe1990Censusandstemmedthedeclinethathadbeenthe D  trendoverrecentdecades. *z  TheNRFUeffortwascompletedaheadofschedule. F OrganizationandStructure:  Post1998,theCensusBureauoperatedwithinanorganizationthatwaswellstructuredto  supportitsperformanceobjectives.Thedecennialorganizationwasorganizedbybusiness t processdrawingfromfunctionalcapabilitiesresidingwithintheparticipatingdivisionsas Z required.Withinthedecennialcensusbusinessprocess,asingleprocessowner,theAssociate @ DirectorforDecennialCensus,controlledbothlineandfundingauthorityfordeliveringthe &v decennialcensusproduct.TheoverallCensusBureauorganizationalstructureislogical,clearly  \ defined,and,forthemostpart,tailoredtowardachievingdesiredresults. B Inmanyofthesubstructuresandteamswithinthedecennialorganization,however,theleaders  oftheteamsanddecisionmakingbodieswerenotgivenordidnotchoosetoexercisetrue  decisionmakingauthority.Leadersoftenservedexclusivelyasfacilitatorsandconsensus   builders.Althoughtheintentbehindthecreationoftheseorganizationalbodieswastopush p!  decisionmakingtothelowestmanagementlevelstechnicallypossible,therewasnodecision V"! makingauthorityinplaceattheselowerlevelstosupportthatintent. <#" ManagementApproach: %X $ Middecade,theCensusBureauattemptedtoinstitutematrixmanagementtoencourageteaming &$"& andtodistributedecisionmakingtolowerlevels.Thisearlyattemptfailedbecausetherewasno ' #' centralizedintegratingorcoordinatingprocessforcensusplansandoperations,anddecision (#( makingmechanismswerenotproperlyimplemented.After1998,a"centralized"management )$) approachaddedstrongcoordination,integration,anddecisionmakingrolesintheDecennial l*%* ManagementDivisiontotheteamingarrangementsfromthefailedmatrixapproach.The R+&+ centralizedapproachfulfilledthedesiredresultsofmatrixmanagementbecauseitfitthe 8,', matrixedbusinessmodelofthedecennialorganization.Thecentralizedmanagementapproach  istheapproachthatbestrepresentstruematrixmanagementbecauseitfeaturesamorerobust  DecennialManagementDivisionontheyaxiscounterbalancingandcoordinatingtheinterestsof | andthecapabilitiesresidentwithintheparticipatingdivisionsonthexaxis. b Despiteoperationalsuccesses,theevaluationofthemanagementapproachrevealedareasthat . ~ warrantimprovement.Inassessingthemanagementmodel,itisimportanttoconsidertwo  d externaldriversthatCensusBureaumanagementmustrespondto(politicalinfluencesand  J technologicaladvances),butoverwhichtheCensusBureaudoesnothavesignificantcontrol.  0 Duringthe1990s,thecongressionaldebateovertheuseofasamplingstrategyversustraditional    enumerationstrategycausedsignificantchallengestotheCensusBureau.    Asthedebateoverstrategycontinued,fundinglevelslimitedthenumberofstaffmembers x  workingonCensus2000.ThisconstrainedtheCensusBureausabilitytogeneratedetailed ^  implementationplansforeitherasamplingortraditionalenumerationapproach.Thefunding D  shortfallsandaninabilitytomaintainacoredecennialstaffthroughoutthedecadealsoledto *z  criticalweaknessesinthedecennialcensusknowledgebaseandhadramificationsformany ` processesandproceduresbeingexecutedduringCensus2000. F CommunicationsandKnowledgeManagement:  Thereislittlerecordofaninternalcommunicationsapproachortheimplementationofaformal  communicationsstructureforCensus2000. t Aknowledgemanagementcapabilitytoretaincorporateknowledge,tosupportresponsesto @ externalreportingrequirements,andtocommunicateprogrammaticchangestodecennialcensus &v participantsinatimelymannerwouldassistinimprovingcommunicationsandinstabilizingand  \ maintainingthedecennialcensusknowledgebasethroughoutthedecade. B AlthoughtheCensus2000centralizedmanagementapproachisanappropriateapproachthat  matchesthedecennialcensusbusinessmodel,improvementscouldbemadethatwouldmakethe  implementationoftheapproachmoreeffective.Recommendationsforimprovementinclude:   " i"   ilT23  0    Clearlydefineandcommunicatetoalldecennialcensusstafftherolesand V"! responsibilitiesofprogrammanagers. ilTT݌<#"(#(# Ќ  " i"   iU23  0    Developprogrammasterplansdescribingoperationalplansearlierinthecensuscycle %X $ thanwasaccomplishedduringCensus2000. iUU݌%>!%(#(# Ќ  " i"   iV23  0    Developriskmanagementplansasanintegralcomponentofcensusplanning. iVW݌' #'(#(# Ќ   )$)  CategoryRAutomationofCensusProcesses   SummariesofthetwelveevaluationsintheAutomationofCensusProcessescategoryfollow. |  EvaluationR.1.a,TelephoneQuestionnaireAssistanceSystemRequirementsStudy (Titan H SystemsCorporation2001a) . ~ TQAwasalargescaleprogramthatprovidedtelephoneassistancetothepublicduringCensus  J 2000.Anetworkof22callcentersusedacombinationofautomatedtechnologiesandagent  0 responsestohandlecallsfromhouseholdswithinthe50states,DC,andPuertoRico.Thisstudy    presentsinformationbasedondebriefingswithpersonnelinvolvedwiththeTQAprogram.A    separatecustomerservicesurveyevaluation(EvaluationA.1.b)providestheuserperspectiveof   thissystem. x  TQAwasthefirsttimethattheCensusBureauoutsourcedacallcenteroperation.Theprogram D  wasconsideredthelargestoperationofitskindimplementedinthecallcenterenvironment.The *z  systemwasdesignedtoaccommodate11millioncallsandreceivedslightlyover6million ` throughoutitsoperation,fromMarch3,2000throughJune30,2000.Anoutboundservicefor F CEFUstartedwhileTQAwasstillinoperation.Thisprogramusedsomeofthesame , technologiesandasubsetofthecallcenters.  AnumberofissuesconfrontedtheTQAdevelopmentteamsuchasthehighcallvolume,call  distributionacrosscenters,therangeofquestionsandtopicsthatwouldrequireresponses,and t thecompletionoftheautomatedshortformquestionnairebycallcenteragents.Despitethe Z challengesandshorttimeframefordevelopment,theprogramwasconsideredextremely @ successful.BothElectronicDataSystemsandCensuspersonnelprovidedexceptionalsupport &v anddedicationtoensurethatTQAwassuccessfullyimplemented.  \ Compromisesonsomerequirementstocollectcertaindataweremadebecauseofthelimited ( timetodevelopthesystem.Althoughmanycompromisesweremadethroughnegotiationswith  subjectmatterexpertsandprogrammanagers,thelackofsomedatafromTQAimpactedthe  completenessofpostCensusevaluations.Somerequirementsweredroppedbecauseoftimeline   andresourceconstraints,butthemajorityofthemissingmanagementandevaluationdatacanbe p!  attributedtotheGeoTelIntelligentCallRouter. V"! TheprovisionofrequireddatathroughtheIntelligentCallRouterprovedtobeasignificant "$r# deficiency.Therequirementsforperformanceandevaluationdataapparentlyexceededwhatthe %X $ IntelligentCallRoutertypicallyprovides.TheCensusBureaurequirementsnecessitated %>!% customizedprogrammingwhichapparentlyexceededthesystemstechnicalcapabilities.Itwas &$"& alsonotclearthatthetechnicalsupportinvolvedinprogrammingtheapplicationcompletely ' #' understoodtheimplicationsofthoserequirements.Theproductwasintendedtomeetthecall (#( routingandinformationtrackingrequirementsthathadbeenclearlyspecifiedbytheCensus )$) Bureau,buttheproductfellshortofitsexpectationsandeitherdidnotprovidethedataor l*%* providedthedatatoolateintheprocess. R+&+  8,', Theseandotherfindingsledtothefollowingkeyrecommendations:   * #$%&'((xx * #$%&'(" i"   ig23  0    Begindevelopmentearly. ig h݌|(#(# Ќ  " i"   ih23  0    Establishrealisticdevelopmenttimelines.Foreachdevelopmenteffort,timelinesmust H considerthecomplexitiesassociatedwithtranslatingandimplementinghighleveluser . ~ requirementsintoafunctionalsystem.Inaddition,timefortestingandreworkis  d requiredtoensurethateachsystemissufficientlystableforproduction. ihh݌ J(#(# Ќ  " i"   ij23  0    Establishagencywideguidance. ijk݌  (#(# Ќ  EvaluationR.1.b,CoverageEditFollowupSystemRequirements (TitanSystems   Corporation2001b) x  TheCEFUprogramwasalargescaleeffortdesignedtoprovideoutboundcallingservicesfrom D  anetworkof13callcentersasameanstoresolvecoverageeditfailures.Thesystemwas *z  developedwithinaverystricttimeframeusingmanyofthesameCensusBureauandcontractor ` resourcesdevotedtothedevelopmentoftheTQAprogram.Thisstudypresentedinformation F basedondebriefingsofpersonnelinvolvedintheCEFUprogram. , Census2000wasthefirstuseofoutboundcallingcombinedwithanautomated,scripted  instrumenttocollectthenecessarycensusdata.Anestimated3.1millioncaseswereanticipated t forthesystemwith2.36millioncasesactuallyidentifiedforfollowup.Theprogram Z commencedonMay8,2000andwasscheduledforcompletiononJune15,2000.Telephone @ followupwasextendedthroughAugust13,2000topermitfollowupwithlanguagedifficulty &v casesandtomaximizethenumberofhouseholdsforwhichtheCensusBureaucouldobtaina  \ completedinterview.CEFUhada70.8percentcompletionrateforcaseswithavalidtelephone B number. ( ThecallcenterinfrastructureandautomatedfeaturesusedforTQAprovidedthefoundationfor  theimplementationofCEFU.Itwasknownthatsomeformofoutboundcallingserviceswould   beusedinCensus2000whentheTQAcontractwasawardedtoElectronicDataSystems; p!  however,thescopeandspecificrequirementsfortheprogramwerenotdefineduntilverylatein V"! thedevelopmentprocess.Giventhetimeframefordevelopment,notallrequirementswere <#" implementedandtestingwaslimited.Regardlessofthetremendouspressuresandissues "$r# associatedwithitsdevelopment,thecontractorandgovernmentpersonnelprovidedexceptional %X $ supportanddedicationtoensurethatCEFUwassuccessfullyimplemented. %>!% CompromisesonevaluationdatarequirementsforCEFUweremadebecauseofthelimited ' #' developmenttime.Althoughthesecompromisesweremadethroughnegotiationsbetween (#( subjectmatterexpertsandprogrammanagers,thelackofsomedatafromtheoperationimpacted )$) thecompletenessofEvaluationI1,CoverageEditFollowupforCensus2000. l*%*   R+&+ Theseandotherfindingsledtothefollowingkeyrecommendations:  " i"   i w23  0    Begindevelopmentearly. i w9w݌|(#(# Ќ  " i"   iw23  0    Establishrealisticdevelopmenttimelines.Foreachdevelopmenteffort,timelinesmust H considerthecomplexitiesassociatedwithtranslatingandimplementinghighleveluser . ~ requirementsintoafunctionalsystem.Inaddition,timefortestingandreworkis  d requiredtoensurethateachsystemissufficientlystableforproduction. iw x݌ J(#(# Ќ  " i"   iz23  0    Establishagencywideguidance. izJz݌  (#(# Ќ  EvaluationR.1.c,InternetQuestionnaireAssistanceSystemRequirementsStudy (Titan   SystemsCorporation2001c) x  IQAanditscompanionsystem,IDC,wereInternetbasedsystemsthatweredevelopedbythe D  sameteam.Whiletherewasatechnicallinkagefromanoperationalstandpoint,forpurposesof *z  thisreporttheywillbetreatedasseparatesystems.Thisstudypresentedinformationbasedon ` debriefingswithInternet F QuestionnaireAssistancesystemdesigners.Aseparatecustomerservicesurveyevaluation , (EvaluationA.2.c)providestheuserperspectiveofbothsystems.  TheoverallobjectiveofIQAwastoprovideinformationtothepublictoassistrespondentswith  completingcensusquestionnaires.Thiswasaccomplishedthroughauserfriendlyinterface t whichprovidedhyperlinkscoveringavarietyoftopics.Languageassistanceguideswerean Z importantfeatureofIQAandcouldbeeasilydownloadedbynonEnglishspeakingrespondents. @ IQAwasassessedbyCensusBureaupersonnelasbeingverysuccessful.Oneofthegoalsofthe  \ systemwastolessentheburdenontheTQAoperation.TheIQAwebsitereceived23,864,598 B hitsbetweenMarch3,2000andApril19,2000.Thesystemwasdevelopedunderavery ( aggressivescheduleanddeployedwithin18months.Theshorttimeframewasduetoan  interruptionoftheplanningeffortthatstemmedfromhighlevelconcernsoverthesecurity  implicationsofIDC.   IQAwaspositivelyperceivedbythepersonnelinterviewedas"therightsystemforthejob."It V"! appearedtoprovideaneffectivemeansofdisseminatingCensus2000informationandforeign <#" languageassistanceguidestothepublicthroughauserfriendlyinterface.Thefewsystem "$r# shortcomingswereallminor.Futureversionsofthesystemwouldbenefitfromenhancements %X $ suchasemailresponseorotherfeedbackmechanisms. %>!% Onemajorrequirementanddesignissuewastheneedforthesystemstoprovidestronglevelsof ' #' securityagainstunauthorizedintrusion;IQAsucceededinprovidingthisprotection. (#( Thesuccessofthesystemwaslargelyduetotheinvolvementofafewhighlytalentedand l*%* dedicatedCensusBureaupersonnel.Contractorsupport(withtheexceptionofpenetration R+&+ testing)didnotplayaroleintheplanningordevelopmentofthesystem. 8,', ЇTheseandotherfindingsledtothefollowingkeyrecommendations:  " i"   ix23  0    Establishagencywideguidance. ix݌|(#(# Ќ  " i"   iN23  0    Beginearlyplanningfor2010.WidespreadInternetusagewillplacesignificantdemands H onthenextgenerationquestionnaireassistancesystem.Therefore,itisrecommended . ~ thatplanningcommenceearlyandreflectanexpectationofgreaterusagerequirements. iN{݌ d(#(# Ќ  " i"   i323  0    Useflexibleprojectcontrolstructure.Theuseofanoversightbodytoguideandmonitor  0 systemdevelopmentactivitiesisconsidereda bestpractice;however,anyproject    controlstructureshouldbeflexibleenoughtoencouragetechnicalinnovation. i3`݌  (#(# Ќ  " i "   i23  0    Assessstaffingrisks.Historically,theCensusBureaureliedoninhouseexpertiseto x  developdecennialsystems.InthecaseofIQA,asingleindividualwasthedrivingforce ^  behindthedevelopmenteffort.Giventhenature,scope,andcomplexityoffuture D  systems,therearerisksinherentinrelyingsolelyoninhousestaffwithout *z  supplementingtheseresourceswithexternalsupport. iB݌`(#(# Ќ  EvaluationR.1.d,InternetDataCollectionSystemRequirementsStudy (TitanSystems , Corporation2001d)  CensusBureaupersonnelassessedIDCasbeingverysuccessfulinspiteoflimitedusebythe  public.Theoverallobjectiveofthesystemwastoprovidecensusrespondentswithahighly t secureInternetfilingoption.IDCsucceededinreplicatingthepapershortformskeyfeatures. Z Softwarelimitationspreventedprecisereplicationofthepaperformineverydetail,however, @ thisdidnotimpacttheform'susefulness. &v OneofthegoalsoftheIDCsystemwastoeasethedatacaptureburdenontheDCS2000; B however,itdidnotmeetthisexpectationasonlyabout66,000formsweresubmittedthrough ( IDCinMarchthroughApril2000.Duringthistime,systemutilizationwascharacterizedbytwo  verybriefperiodsofheavyaccess.IDCdidnotachieveconsistentandwidespreadusageowing  toalackofpublicitythatstemmedfromInternetrelatedsecurityconcernsatseniormanagement   levels.Thesystemwasdevelopedunderaveryaggressivescheduleanddeployedwithin18 p!  months V"! IDCwaspositivelyperceivedbythepersonnelinterviewedas therightsystemforthejob.It "$r# providedaneffectivemeansofcapturingCensus2000informationthroughthesubmissionof %X $ shortformquestionnairesovertheInternet.Thefewsystemshortcomingswereallminor. %>!% However,thereweresomeusabilityproblemsthatwerenotresolved. &$"& Onemajorrequirementanddesignissuewastheneedforthesystemtoprovidestronglevelsof (#( securityagainstunauthorizedaccess;IDCsucceededinprovidingthisprotection.This )$) requirementinterferedwithincludingmembersofthepublicinusabilitytesting. l*%*   R+&+ SimilartoIQA,thesuccessoftheIDCsystemwaslargelyduetotheinvolvementofafew  highlytalentedanddedicatedCensuspersonnel.Contractorsupport(withtheexceptionof  penetrationtesting)didnotplayaroleintheplanningordevelopmentofthesystem. | Theseandotherfindingsledtothefollowingkeyrecommendations: H " i "   i23  0    Establishagencywideguidance. i݌ d(#(# Ќ  " i "   i23  0    Beginearlyplanningfor2010.IDCwasessentiallyoverengineeredforcapacityand  0 redundancybecausetheanticipatedtrafficlevelsdidnotmaterializeduetoalackof    publicity.WidespreadInternetusewillplacesignificantdemandsonthenextgeneration    datacollectionsystemin2010.Therefore,itisrecommendedthatplanningcommence   earlyandreflectanexpectationofradicallydifferentcapacityandredundancy x  requirements. i݌^ (#(# Ќ  " i "   i`23  0    Assessstaffingrisks.Historically,theagencyhasreliedoninhouseexpertisetodevelop *z  decennialsystems.Giventhenature,scope,andcomplexityoffuturesystems,thereare ` risksinherentinrelyingsolelyoninhousestaffwithoutsupplementingtheseresources F withexternalsupport. i`݌,(#(# Ќ  EvaluationR.2.a,OperationsControlSystem2000SystemRequirementsStudy (Titan  SystemsCorporation2002a)  TheCensus2000OperationsControlSystem(OCS2000)wasperceivedasasuccessfulsystem Z thatwasdeployedwhenneeded,inspiteofthelackofastandardizedrequirementsdefinition @ processandthesubstantialchangesinrequirementsthatoccurredduringthelifeofthesystem. &v Theoverallobjectiveofthesystemwastoautomatethemanagementoffieldoperationspriorto  \ andduringCensus2000.Thisstudypresentsinformationbasedondebriefingswithpersonnel B involvedwiththeOCS2000program. ( Thesystemassignedandcontrolledworktoallcensusenumerators,trackedprogressof  assignments,producedcostandprogressreportsonfieldoperations,printedawidevarietyof   enumerationrelatedmaterials,andassistedwiththemanagementandtrackingofshipping p!  documents.TheOCS2000wasoperationalbetweenOctober1997andAugust2000andhadsix V"! keyinterfaceswithothersystems,oneofwhichwasexternaltotheCensusBureau(theFedEx <#" interface).ItprovidedsupportforseveraloperationsincludingNRFU,thelargestsinglefield "$r# operationinCensus2000. %X $ ItssuccesswasparticularlynoteworthyinviewoftheimpactofaSupremeCourtdecisionwhich &$"& changedthefocusofthedecennialcensusfromasamplingbasedapproachtofullenumeration. ' #' Thedecisionwashandeddownverylateinthesystemdevelopmentcycle.Althoughthe (#( DecennialManagementDivisionprovidedaprojectoversightroletoensurethattheright )$) resourceswereappliedtotheproject,uncertaintyoverwhichmethodtheSupremeCourtwould l*%* favorresultedinadualsystemapproach(i.e.,samplingandfullenumeration)todevelopment R+&+ goingintotheDressRehearsalin1998. 8,', ЇPrimarysystemdeficienciescenteredaroundtwoareas:theneedforaFedExinterfaceand  difficultiesencounteredwithobtainingtestdata.TheFedExinterfaceworkedmoderatelywell;  however,onceapackagewasshipped,thetrackingdatawerenotavailableintheOCS2000. | ThisrequiredoperatorstogototheFedExwebsitetomonitorshippingstatus.Datato b sufficientlytestinterfacestoothersystemswerenotalwaysavailable.Thislackofdatawas H causedbecausesomesystemswerestillunderdevelopmentandsomeoperationsinthefield . ~ werenotyetcomplete.  d Aperceivedneedforexpandedsystemaccessandnationallevelstatusreportswasnotedby  0 someusersduringtheinterviews.Suchcapabilitiescouldhavebeenimplementedbutwere    problematicandinconsistentwithfundamentalsystemobjectives.Theinclusionofthese    capabilitieswouldnothaveimprovedoverallsystemfunctionalityinameaningfulway.   Theseandotherfindingsledtothefollowingrecommendations: ^  " i "   i 23  0    Establishagencywideguidance. i :݌*z (#(# Ќ  " i"   i23  0    Conductcontractororientation.Theagencyutilizedcontractorsupporttodevelopthe F OCS2000.Programminganddatabasefunctionswereperformedbycontractorsworking , inhouseunderthedirectionoftheTechnologiesManagementOffice.Thesecontractors  werewellintegratedwiththestaff,werehighlycompetent,andperformedextremely  well.However,theydidnotalwaysunderstandbusinesspracticesattheCensusBureau. i݌(#(# Ќ  " i"   i23  0    Assigncontractingofficertoteam.Theassignmentofadedicatedcontractingofficerto Z theOCS2000teamprovedtobeaneffectivearrangementthatfacilitatedthetimely @ handlingofcontractualissues.Assigningacontractorofficeraspartoftheoverallteam &v isabestpracticethatshouldbeconsideredforlargeand/orcriticalsystemdevelopment  \ projects. iĮ݌B(#(# Ќ  EvaluationR.2.b,LaptopComputersforAccuracyandCoverageEvaluationSystem  RequirementsStudy (TitanSystemsCorporation2002b)  Theuseoflaptopcomputerstocollectrespondentdatawasperceivedasahighlysuccessful p!  platformforthePersonInterviewphaseoftheA.C.E.program.Alaptopbasedautomated V"! questionnairewasusedbyinterviewerstoconductpersonalandtelephoneinterviewstoassist <#" withthedeterminationofcoverageerror.Inaddition,laptopsservedasaplatformforacase "$r# managementsystemandprovidedremotemailservices.Thisstudypresentsinformationbased %X $ ondebriefingswithpersonnelinvolvedwiththeLaptopsforA.C.E.program. %>!% ThereweretwoautomatedinstrumentsforComputerAssistedPersonInterviewing(CAPI)for ' #' A.C.E.,oneforthePersonInterviewoperationandanotherforthePersonInterviewQA (#( operation.ThelatterwasusedtoconfirmthattheindividualconductingthePersonInterview )$) hadactuallycontactedtheoriginalrespondent.Forcaseswhentherespondenthadnotbeen l*%* contacted,thePersonInterviewQAinstrumentcontainedacompleteversionofthePerson R+&+ Interviewtherebyenablingtheinterviewertocollectthenecessaryinformation. 8,', ЇTheCensusBureaubuiltuponitsexperienceusinglaptopcomputersforsurveydatacollection  thatextendedbackasfaras1992.ThelaptopprogramforCensus2000effectivelyutilizeda  customizedversionofanexistingquestionnaireauthoringsoftwarepackagetodevelopthe | questionnairefortheA.C.E.program.Thelaptopsprovidedahardwareplatformforalogic b basedinstrumentthatguidedtheinterviewerthroughthedatacollectionprocess.Interviewers H likedusingthelaptop.Asidebenefitofthelaptopswasthattheylentaprofessionalappearance . ~ totheinterviewersthatmayhaveservedtoreduceconcernsoverthereleaseofpersonaldataby  d interviewees.  J Thereweredauntinglogisticalandsupportissuesrelatedtousinglaptopswithautomatedsurvey    instrumentsfordatacollection.Nevertheless,theCensusBureauoptedtousethemasthe    platformfortheA.C.E.datacollectionoperations.Manyofthoseissueswereformidable.For   example,theCensusBureauhadtoacquire,configureanddeployover9,000laptopsnationwide. x  Oncedeployed,thereweremajoraccountability,training,andmaintenanceissues.Inspiteof ^  thecostsandproblemsposedbytheseissues,theCensusBureausdecisiontousethelaptops D  provedtobeagoodone. *z  Thelaptopswereaveryeffectivetoolandperceivedastherightplatformforthejobinthatthey F greatlyfacilitatedcasemanagement,acceleratedthedatacollectionprocess,andimproveddata , quality.Oneuniquefacetofthelaptopprogramwasitsabilitytoexchangedatawith  HeadquartersandROsviaaremotedialintelecommunicationssession.Thismethodwasvery  fast,secure,andreliable.  Offpathdataaregeneratedwhenaninterviewerneedsto"backup"tomakeacorrectiontothe Z dataentered.Thisaction,insomecases,necessitatesashiftintoanotherlogic"path"pertaining @ tothecorrectresponse.Offpathdatawereidentifiedasarequirementintheplanningphase;the &v ComputerAssistedSurveyExecutionSoftwarewasdesignedtoeithergloballykeepoffpath  \ dataortoignoreitentirely.Offpathdatawereimportantbecausethesedatawerenotalways B incorrect;instead,thedatamayhaveresultedfromalegitimateinterview.Procedureswere ( establishedtodistinguishbetweenthedatathattheCensusBureauwantedtokeepandthosedata  thatwerenotimportant.However,theseprocedureswerenotcorrectlyimplemented.Sinceall  offpathdatawerecaptured,theCensusBureauwasabletoperformpostcensusprocessingedits   torestoreanymissingvalues. p!  Theseandotherfindingsledtothefollowingkeyrecommendations: <#" " i"   i23  0    Begindevelopmentearly. i݌%X $(#(# Ќ  " i"   iq23  0    Considerfullandopencompetition.Anopencompetitionamongvendorsmayrequire &$"& substantialtimeandeffortintheshortrun;however,thecompetitiveprocessusually ' #' servestomitigaterisksinthelongrunbyassuringthatthevendorhasthenecessary (#( capabilityandexperiencetomeetprojectrequirements. iq݌)$)(#(# Ќ  " i"   i23  0    Identifydataexchangerequirementsearly.TheCensusBureauconsideredthe R+&+ requirementsfortransmittingdataearlyinthelaptopsprogram.Theearlyidentification 8,', oftherequirementshelpedtoensurethetimelinessandaccuracyoftheinformationbeing  transmittedandservedtomaximizenetworkandmachineresourcesbytransmitting  duringoffhours. i݌|(#(# Ќ  EvaluationR.2.c,AccuracyandCoverageEvaluation2000SystemRequirementsStudy  H (TitanSystemsCorporation2002c) . ~ The2000A.C.E.controlsystem(ACE2000)includedbothtrackingandcommunication  J functions.ThesystemwasusedtocontrolA.C.E.fieldoperationsandsomesmalleroperations  0 suchasrelisting,TargetedExtendedSearch,andQA.A.C.E.productionusagebeganinAugust    1999andcontinueduntilMay2001.Thisstudypresentedinformationbasedondebriefingswith   personnelinvolvedwiththeACE2000system. x  Theoriginaltrackingandcontrolsystemwascharacterbased,operatinginaDiskOperating D  System(DOS)environment.AWindowsbasedversionofthesoftwarewasdevelopedaspartof *z  theIntegratedCoverageMeasurementprogram.TheIntegratedCoverageMeasurement2000 ` systemwasusedtocontrolandtrackallIntegratedCoverageMeasurementfieldoperationsin F additiontotheComputerAssistedPersonalInterviewoperationsduringtheCensus2000Dress , Rehearsal.AlthoughtheDressRehearsalwasconsideredasuccess,issuessuchasdatavolume,  changingsurveyrequirements,andconcernsregardingperformanceresultedintheneedfor  furthersystemenhancements.IntegratedCoverageMeasurement2000wasrenamedACE2000  whenthesurveywasrenamedA.C.E. t TheACE2000systemwasconsideredbythoseinvolvedinthestudytobethe rightsystemfor @ thejob.Thesystemwassuccessful,butitssuccessresultsfromthededicationoftheCensus &v Bureauandcontractorstaff,notfromawellplanneddevelopmenttimelineorsupporting  \ developmentmethodology. B Thedevelopmentofthesystemwassubjecttocontinuouschanges.Thechangesforcedthe  developmentteamtofocusoncorefunctionalitythuspreventingsomerequirementsfrombeing  implementedduetoscheduleconstraints.   Thenumberofchangeslimitedthetestingthatcouldbeaccomplished.Althoughseverallayers V"! oftestingwereinplace,testingwasimplementedbasedonacarefulreviewofresources,time <#" available,andrisk. "$r# Theseandotherfindingsledtothefollowingrecommendations: %>!% " i"   id23  0    Defineadequateresources.DevelopmentoftheACE2000systemwassubjectto ' #' wholesalechangesincensusmethodology,technology,andbusinessprocess.Acensus (#( isnotthetimetotryunprovenapproachesortodevelopsystemswithoutsufficienttime )$) fordevelopmentandasufficientlylargestaffofbothsubjectmatteranddevelopment l*%* personnel.TheCensusBureaumustplanwellinadvancetoensurethatthenecessary R+&+ personnelresourcesareavailabletosupporttheprojectandthatthoseresourcescan 8,', devoteasufficientamountoftimeinrequirementsdefinitionandtestingwithoutbeing  divertedtootheractivities. id݌(#(# Ќ  " i"   i23  0    Replicatecensusenvironmentmoreclosely.TheCensus2000DressRehearsal b environmentprovidedforonlyoneLCOwithineachofthethreeROsinvolvedinthe H exercise.Thisenvironmentdidnotprovidetheopportunitytotestandevaluatecertain . ~ aspectsofthesoftware.RecommendationsincludethattheCensusBureauestablisha  d morecompletecrosssectionofthebusinessprocessduringthedressrehearsalsothatthe  J nuancesofeachoperationcanbebettertestedandevaluated. iE݌ 0(#(# Ќ  EvaluationR.2.d,MatchingandReviewCodingSystemforAccuracyandCoverage    Evaluation (TitanSystemsCorporation2002d)   TheMatchingandReviewCodingSystemwassoftwaredesignedtofacilitateclericalmatching ^  ofrecordsbetweencensusdataandA.C.E.data.Thisstudypresentedinformationbasedon D  debriefingswithpersonnelinvolvedwiththeMatchingandReviewCodingSystem. *z  AcharacterbasedsystemwasdevelopedbyinhouseresourcesandusedfortheCensus2000 F DressRehearsal.Thissystemlackedsufficientfunctionalitytomeettheneedsofthematching , process.Adecisionwasmadetooutsourcedevelopmentofanewsystemthatcouldprovidethe  pointandclickinterfaceofWindowsbasedsoftware.TheMatchingandReviewCoding  softwarewasdeveloped,successfullydeployed,andconsideredbythoseinvolvedinthestudyto  bethe rightsystemforthejob.Althoughsomerequirementschangedovertimeandnew t requirementswereintroduced,thededicationofthecontractorandagencypersonnelresultedin Z asuccessfulsystemdevelopmenteffort.Thiswasaccomplisheddespitesignificanttimeand @ resourceconstraints. &v Thesoftwarewasdevelopedasthreedistinctsystems.ThisenabledCensusBureaupersonnel B anddeveloperstobuildupontheirexperiencesandapplycriticallessonslearnedduringthe ( developmentprocess.Althoughthesamegeneralprocesswasusedtoidentify,clarify,and  implementrequirementsacrossthesystems,theproceduresandmeansofcommunicationwere  finetunedovertime.   Thesoftwareprovidedpointandclickfunctionalitytofacilitatetheclericalmatchingprocess. V"! Thesoftwarewasmorestreamlinedthanearliersystemsallowingtheclerkstomovefaster <#" throughthematchingprocess.Technologicalimprovementssuchassplitscreenandfilter "$r# capabilitiesalsoenhancedthesystemsusability. %X $ Theseandotherfindingsledtothefollowingkeyrecommendations: &$"&   " i"   i23  0    Conductteamtraining.Ateamapproachwasutilizedtoidentifyrequirementsand (#( evaluatethesoftware.Ateamapproachenablestheorganizationtosolicitdifferent )$) viewpointsandensurethatthewidestrangeofneedsareaddressed. i݌l*%*(#(# Ќ   8,', " i"   i23  0    Clarifyrolesandresponsibilities.Severaltestingprocesseswereimplementedbetween  thecontractor,alphatestgroups,andtheCensusBureauBetaTestSite.Caremustbe  takentoclearlyidentifytherolesandresponsibilitiesofeachgroupandproceduresmust | beestablishedtoensureaneffectiveandefficientmeansofsharingtestresults.Thiswill b helpensureconformancewithrequirementsspecificationsandcomprehensivecoverage H oftestingneeds. i݌. ~(#(# Ќ  EvaluationR.3.a,PreAppointmentManagementSystem/AutomatedDecennial  J AdministrativeManagementSystemSystemRequirementsStudy (TitanSystems  0 Corporation2002e)    ThePreAppointmentManagementSystem/AutomatedDecennialAdministrativeManagement   System(PAMS/ADAMS)wasthefirstfullyintegratedapplicant,personnel,andpayrollsystem x  developedfortheDecennialCensus.Thisenterprisewidesystemutilizedstateoftheartclient ^  servertechnologytomanageanddistributedatato12RCCs,thePuertoRicoAreaOffice,and D  520LCOs.Theoverallobjectivewastodevelopacomprehensivesystemfortemporary *z  employeesthatmanagesthecompleteemploymentlifecycle.Thesystemwassuccessfulat ` integratingprocessesthatwerepreviouslyhandledbynoninteroperable(i.e.,independent) F systems.Timeconstraintsimposedbylatefundingoftheprojectplacedlimitsonthe , implementationofsomerequirements,howeverthisdidnotaffecttheoverallperformanceofthe  system.Thisstudypresentsinformationbasedondebriefingswithpersonnelinvolvedwiththe  PAMS/ADAMS.  ThePAMS/ADAMSwasquitelargeinscope.Inall,therewereover3,000,000applications Z processedandweeklypayrollreachedamaximumofover520,000individuals.Previous @ systemsweredevelopedinhouse.Acommercialofftheshelfproductthatcouldfullymeetthe &v uniqueneedsofCensus2000wasnotfound,howeverthecommercialproductwiththebestfit  \ wasusedasthebasisfordevelopment. B Aformalizedmethodwassuccessfullyusedtoidentifyrequirements.However,therewere  significantobstaclestoimplementinganeffectivesystem,especiallypriortothedressrehearsal.  Thedevelopmentofthesystemwasimpactedbyrequirementschangesandperformanceissues.   Althoughthereweresignificantchallengesthatposedhighrisksthroughoutthedevelopment p!  process,theproductionsystemperformedtothesatisfactionoftheCensusBureauandits V"! stakeholders. <#" Althoughthesystemproducedmanytypesofreports,otherrealtimereportingrequestswerenot %X $ fulfilledbecauseoflatedatawarehouseimplementation,inabilityofmanagementreporting %>!% systemstofullyusefeedersystemdata,andheavyprogrammerworkload. &$"& PAMS/ADAMSincorporatedauserfriendly,locallyoperatedfrontendinterfaceforcapturing (#( applicantandpayrollforms.Itwasdevelopedtoaddressdatacaptureissuesthataroseafterthe )$) originalscanningrequirementwasdropped.Thisinterfacewasdesignedforawidevarietyof l*%* users,takingintoaccountthelevelofeducation,physicallimitationsandtheabilitytominimize R+&+ humanerror.Thiswasamajorfactorthatcontributedtothesystemsefficiencyandsuccess. 8,', ЇThesuccessofthesystemwaslargelyduetothehighlydedicatedteamofCensusBureauand  contractorpersonnel.Thisexperienced,cohesiveteamworkedtogethertoaddresstechnicaland  proceduralissuescapably,evenunderthepressuresassociatedwithtimeconstraintsand | mandatorydeadlines.Allmajorperformanceissueswereresolvedbeforedeployment. b )X XXXzTheseandotherfindingsledtothefollowingkeyrecommendations:#XzX X)X`# . ~ " i"   i23  0    Designforflexibility.Thesystemsprocesseswerehighlyregulatedandtherefore  Q subjecttochangeinlocalandnationallawswhichaffectedsystemrequirements.The  7 developmentteamfounditdifficult,attimes,toimplementthesepolicychangesinthe    system.Thedesignofthesystemoughttobemodularizedandbeadequatelysizedand    flexibleenoughtoaccommodatethesetypesofchanges. i'݌ (#(# Ќ  " i"   i23  0    Includeallstakeholders.Thesystemproducedmanyautomatedandadhocreportsbut e  thereweremanyrealtimereportingrequeststhatwerenotfulfilled.Reporting K  requirementsneedtobeconsideredfrommanydifferentviewpointsandstakeholders 1  fromallkeyareasshouldberepresentedduringthisphasetoensurethattheir g informationneedsaremet.Identificationofreportingneedsduringtherequirements M phasewillmaximizethebenefitthatcanbederivedfromthesystemforallusers. i݌3(#(# Ќ  " i"   i23  0    Implementformalprocess.Modificationswereperformedcontinuouslyandanefficient  formalchangecontrolprocesswasemployed.Additionally,awelldocumentedSystem  InvestigationsRequirementlogwasusedthroughouttheentirechangemanagement { process.Changesshouldbesystematicallyassessedinlightofprogrammaticgoals.The a requirementsforchange&  controlandsupportingdocumentationshouldbeincludedinthe G systemdevelopmentmethodology.'GԀTheChangeControlBoardalsomusthaveadequate -} resourcestoaddressprogramswithlargeandcomplexscopes. i݌c(#(# Ќ  EvaluationR.3.b,AmericanFactFinderSystemRequirementsStudy (TitanSystems / Corporation2002f)  AmericanFactFinder(AFF)isanInternetenabledinformationsystemwhichprovidesan   efficientmeansofmakingawiderangeofcensusinformation(demographic,economic,and w!  geographic)availabletoCensusBureaupersonnelandexternalusers.IDCwasdesignedtomeet ]"! theneedsofallusers,fromnovicetoexpert.Duetothediversityofsystemusers,thesystem C#" interfaceprovidedaninteractiveanduserfriendlywaytofacilitateretrievalanduseof )$y# informationanddata.Thisstudypresentsinformationbasedondebriefingsofpersonnel %_ $ involvedintheIDCprogram. %E!% Acontractorplayedamajorroleindesigning,sizing,andoperatingthesystem.Thedecisionto '#' usecontractorsupportstemmedfromtherealizationthatInternettechnologywasevolving (#( rapidlyandthatoutsideexpertisewasneededtosuccessfullyimplementastateoftheart )$) system.Aniterativedevelopmentprocesswasemployedusingacyclicalbuildingtechnique s*%* (design,build,andtest)thatallowedforcontinuousfeedbackandevaluation.Thecontractor Y+&+ wasalsoapartnerwiththeCensusBureauintherequirementsdefinitionprocess.Theprocess ?,', includedgatheringrequirementsfromkeystakeholders,subjectmatterexperts,andpotential  usersviainterviewsandjointapplicationdevelopmentsessions.Theresultsoftheinterviews  werecompiledandincorporatedintoUseCasedocumentation. | Theprincipalcontractor,IBM,hadtwoyearstodevelop,test,anddeploythesystem.A H subcontractorwithexpertiseingeographicinformationsystemsandmappingapplicationsalso . ~ wasbroughtintosupportthedevelopmentofthesystem.Thefirstproductionimplementation  d ofIDCwasinMarch1999andprovidedaccesstoeconomicdata,theACS,and1990Census  J data.AsecondimplementationcameinDecember2000.Itprovidedimprovedperformance,  0 addressedusercommentsandrequestsconcerningtheuserinterface,andscaledupthesystemto    accommodateworkloadsassociatedwithCensus2000.    AswithothersystemsthatweredevelopedtosupportCensus2000,IDCdidnotbenefitfroman x  agencywidestandardprocessforrequirementsdefinition.Therequirementsmethodologywas ^  providedbythecontractorandtheagencyconductedanalysesofsecurityneedsanduser D  segmentation.Overall,IDCwas therightsystemforthejobinthatitsucceededinproviding *z  aneffective,thoughnotalwayseasytouse,toolthroughwhichmanydifferenttypesofusers ` couldforthefirsttimeaccesscensusdataondemand. F TheneedtoefficientlydisseminateCensus2000datawasamaindriverfordevelopingIDC.  However,thesystemdisseminatesothercensusdatathataregeneratedbyvariousprogramareas  oftheCensusBureau(i.e.,economiccensusesandsurveys,demographicsurveys,andtheACS).  Themultifacetednatureofthesystemandplansforfurtherexpansionrequirethatthissystem t remainactive.Thus,unliketheother11automatedsystemsthatwereevaluatedspecificallyas Z supportingcomponentsofCensus2000,IDCisacorporatesystemnotadedicatedCensus2000 @ system. &v IDChasbeenamajorsuccessfortheCensusBureaufromthestandpointofachievinga B breakthroughinthedeliveryofvoluminousdatainanelectronicformatandinmakingthesedata ( availabletoexternalusers.Italsoachievedareduction(thoughnotelimination)intheuseof  traditionalmedia(printedhardcopy,magnetictape,etc.).Thesystemholdsgreatpromisefor  escapingthelimitationsofpredefinedcensusdatabymakingcustomizedqueriespossible.   Confidentialitywasamajordesignfactorfromtheoutset.Giventheneedtoprohibit V"! unauthorizedaccesstoconfidentialmicrodatafilesandtominimizeopportunitiesforre <#" identification(i.e.,combiningmultipledatasourcesinanefforttoequatecensusdatawith "$r# particularpeople),theCensusBureauhasundertakenprecautionaryandeffectiveeffortsto %X $ ensuresecurityandpreventunauthorizedaccesstodata. %>!% Anevolutionaryapproachwasusedtodevelopthesystemthatrequiredconstantfinetuningas ' #' developmentprogressed.CensusBureaumanagersandprogramstaffwereaware,fromthe (#( outset,thatsystemdevelopmentwouldbeincrementalbecauseofthedeliverycyclesfor )$) DecennialCensus,EconomicCensus,andACSdataproducts.Thus,therewasanunderlying l*%* assumptionforIDCthatthesystemwouldhavetoadapttorequirementsgrowthandthe R+&+ contractingapproachanddevelopmentphilosophy.Changecontrolprocessesweregovernedby 8,', /Oad / /  thisawareness.Changestorequirementswereinitiatedthroughoutthedevelopmentcyclein  keepingwiththeiterativedevelopmentapproach.Adaptingtothechangingrequirementshad  significantcostimplications. | CommunicationbetweenCensusBureauprojectmanagementpersonnelandthecontractorwas H frequent,welldocumented,andincludedaneffectivechangecontrolprocess.Thisprocesswas . ~ especiallyimportantinviewoftheprototypingapproachthatwasemployed.TheCensus  d BureauemployedtheDepartmentofCommerceconceptofoperationsforstreamlined  J acquisitionforitsprocurementmethodology.Thisapproachhelpedtoexploresystem  0 characteristicsanddevelopmentissuesthroughpreaward,facetofacemeetingswithvendors.    Contractorswereencouragedtoutilizecommercialofftheshelfsoftwareasadevelopmenttool    duetosoftwaremaintenanceandotherconsiderations.   Theseandotherfindingsledtothefollowingkeyrecommendations: ^  " i"   ip23  0    Defineuserbaseearly.TheCensusBureaumadeextraordinaryeffortsbetween1995and *z  1997todefinethesystemuserbaseandaddresstheirneedsbyconductingfocusgroups ` withinternalandexternalcustomers,meetingwithprivatesectororganizations, F surveyingparticipantsinvolvedinbetatestingofthesystem,andinterviewingdatausers. , ЀItisrecommendedthattheCensusBureaucontinuethepracticeofconductingcustomer  segmentationanalysesasearlyaspossibleinthesystemdevelopmentprocess. ip݌(#(# Ќ  " i"   i23  0    Establishagencywideguidance. i݌t(#(# Ќ  " i"   ib 23  0    Customizeuserinterfacedesignbyusertype.Thesystemposedamajorchallengeto @ designers/developersinthesensethatitneededtoserveaverydiversesetofusers.Itis &v recommendedthatfuturerefinementsofIDCconsidersettingupuserclassestomakethe  \ systemsuitablefornovicesaswellaspowerusers. ib  ݌B(#(# Ќ  EvaluationR.3.c,ManagementInformationSystem2000SystemRequirementsStudy   (TitanSystemsCorporation2002g)  TheprimarygoaloftheManagementInformationSystem2000systemwastoservethe p!  informationanddecisionsupportneedsofthedecennialcensus.ManagementInformation V"! System2000wasthefirstexecutiveinformationsystemusedbytheCensusBureautoaggregate <#" schedulingandbudgetinformationandwastheofficialsourceofmanagementinformationfor "$r# Census2000.TheManagementInformationSystem2000wasanumbrellasystemwithtwo %X $ components:MasterActivityScheduleandCostandProgress.Thisstudypresentsinformation %>!% basedondebriefingsofpersonnelinvolvedintheManagementInformationSystem2000 &$"& program. ' #' TheMasterActivitySchedulecomponentcontainsinformationontheschedulingandduration )$) ofallcensusactivitiesandwastheofficialentryandupdatepointfordesignatedCensus2000 l*%* data.TheCostandProgresscomponentofthesystemusesanefficiententerprisewidedatabase R+&+ ofcostandprogressinformationdesignedtoenablemanagerstoassessandmodifyoperational 8,', plansastheyrelatetotheMasterActivitySchedule;manageoperationsandcost;andidentify  problemsquickly.  BothcomponentsoftheManagementInformationSystem2000containedabroadvarietyof b functionstosupportthemanagementofthedecennialcensus.Amongthesewereanalytical H toolsusedtoassessprogressandmanagementfunctionstoaidinthedecisionmakingprocess. . ~ Informationwaspresentedinvariousformssuchasreports,graphs,summaries,Pertcharts,and  d schedules.Thesetoolswereusedtomanagetheoperationofthecensus.  J AlthoughtheManagementInformationSystem2000provedtobeausefultoolfordetermining    highlevelcauseandeffectrelationshipswithregardtotheoperationofCensus2000,someusers    indicatedthattheycontinuedtorelyontheirowncontrolsystemsandschedulingtools.These   usersstatedthatthescopeandpurposeofthesystemwereneverclearlydefined.Thebasic x  requirementsastowhatdatawereneededtomonitorprograms,andatwhatlevel,werenot ^  adequatelyaddressed. D  TheCostandProgresscomponentwasnotusedbysomeprogrammanagementofficesand ` divisions.Thiswasdueinparttothecomplexnatureofusingthesoftwareandtheusers F unfamiliaritywithcurrenttechnologysuchasgraphicaluserinterfaces.Regularusagealong , withtrainingwasneededtobecomeproficient.  IntheCostandProgresscomponent,itappearedthatalackofstandardizationbetweensystems  createdproblemsinterpretinginformationinthereports.Someterminologyanddataelement t definitionsweredifferentbetweensystemsandreportscontainingaggregateddataoften Z reflecteddifferentsnapshotsintime. @ Anysystemdesignmustconsidertheentireoperationalenvironmentincludingassociated  \ businessprocessessuchasthosedesignedtomaintainthecurrencyandintegrityofthedata.The B MasterActivitySchedulecomponentwasausefultoolbecauseitprovidedscheduling ( informationatvariouslevelsofdetailthatenabledmanagerstoidentifyissuesandactivitieswith  thepotentialtoimpacttheprojectschedule.Currentinformationrelatedtotaskmanagement  wasnotalwaysavailableasneededbecausetheprocessofupdatingtheschedulewasnot   effectivelyimplemented. p!  Testingservestovalidatethatsystemrequirementshavebeenmet.TestingfortheCostand <#" Progresscomponentwasconsideredcomprehensive.Inadditiontotheunittestingconductedby "$r# developers,anindependentinternaltester,whohadparticipatedintheJointApplication %X $ Developmentsessions,wasassignedtoconductalphatesting.Knowledgeofsystem %>!% requirementsgainedfrombeinginvolvedintheJointApplicationDevelopmentsessionsallowed &$"& thetestertodetermineifrequiredfunctionalityhadbeenimplemented. ' #'   )$) Theseandotherfindingsledtothefollowingkeyrecommendations:  " i"   i623  0    Increaseseniormanagementcommitment.TherequirementsfortheManagement | InformationSystem2000,inparticulartheCostandProgresscomponent,were b developedverylateinthedecennialcycle.Thesystemneededconsistentseniorlevel H managementfocusandsupportthroughoutitsdevelopmentandoperation. i6<6݌. ~(#(# Ќ  " i"   i8823  0    Increasecoordination.ManyofthesystemsfeedingdatatotheCostandProgress  J componentwereproducedindependentlyandusedvaryingdefinitionsfordataelements.  0 ThisimpactedtheabilityoftheCostandProgresscomponenttoreceivedatafromfeeder    systemswithoutreprogramming.Forreliableresults,terminologyanddataelement    definitionsshouldbestandardizedbetweenfeedersystems. i88e8݌ (#(# Ќ  " i"   i:23  0    Providesufficientresources.BothcomponentsoftheManagementInformationSystem ^  2000requiredahighlevelofexpertisetomaintain.Duetoalackoftechnicalsupport D  resources,trainersfortheMasterActivitySchedulebecameinvolvedinotheractivities *z  suchasconfiguringpersonalcomputersandinstallingupgrades.TheCensusBureau ` shouldaddresstheneedforongoingtechnicalsupportduringtherequirementsprocess F andensurethatsufficientresourcesareavailabletosupportdevelopmentteams.This , wouldhelpensurethatdevelopersstayfocusedonthedevelopmentanddeploymentof  thesysteminsteadofdivertingtheireffortstoaddressmoregeneralsupportissues. i::݌(#(# Ќ  EvaluationR.3.d,Census2000DataCaptureSystemRequirementsStudy (TitanSystems t Corporation2002h) Z Census2000DataCaptureprovidedstateofthearthardwareandsoftwaretocapturecensus &v data.Ascanningprocesscreatedadigitalimageofcensusforms;theseimagespassedthrough  \ OMRandOpticalCharacterRecognition(OCR)devicestocaptureinformationelectronically. B Manualkeyingwasusedtoenterdatanotcapturedelectronically.TheCensusBureau ( outsourcedthetwomajorcomponentsoftheCensus2000DataCaptureprogram.Thetwo  componentsweretheDCS2000awardedtoLockheedMartinandtheDataCaptureServices  ContractawardedtoTRW.Thisstudypresentsinformationbasedondebriefingswithpersonnel   involvedinbothcomponentsoftheCensus2000DataCaptureprogram. p!  Census2000DataCapturewasasignificantchallengeinvolvingleadingedgetechnologies; <#" outsourcingofsoftwaredevelopment,hardware/softwareintegration,andoperations;extremely "$r# complexrequirements;andmyriadchanges.Despitethechallenges,theprojectteamwas %X $ successfulinimplementingasystemthatefficientlyandeffectivelyprocessed150million %>!% forms,usinginnovativetechnologiesandcontractingtechniquestoaccomplishthiseffort. &$"& OneofthereasonsforthesuccessoftheCensus2000DataCaptureprogramwasthattheproject (#( teamestablishedacooperativerelationshipwithboththeprimecontractors.Inaddition,the )$) programmanagermadeitcleartoeachorganizationthattheirsuccesswasdependentoneach l*%* other.Thisfactencouragedbothcontractorstoestablishcloseworkingrelationshipsand R+&+ cooperateintheidentificationandresolutionofproblems. 8,', ЇTheprojectteamestablishedastringentchangecontrolprocessattheworkinglevelthatserved  totrack,evaluate,andcontrolchangestotheDCS2000.Thisprocesspleasedprogram  managers,asitmitigatedriskstothedatacaptureprogram.Therequirementsmethodologyand | changecontrolprocessimplementedwerepraisedbyoversightorganizations,suchasthe b GeneralAccountingOfficeandtheInspectorGeneral.Inaddition,TRWusedadocument H managementsystemandemailtotrack,control,andissuechangestoproceduresandtraining . ~ materialsaspartoftheDataCaptureServicesContract.Althoughthestringentchangecontrol  d processwassuccessful,someofthoseindividualsrequiringdatacaptureinformationfor  J evaluationpurposesthoughttheprocesstoorigidinmeetingtheirdatarequirements.  0 AseriesofoperationaltestsanddryrunswereconductedateachoftheDataCaptureCenters.    Thesetestsprovidedanopportunitytoassesstheintegrationofsoftwareandproceduresand   identifiedchangesandimprovementstobothcomponents.Anintegrated FourSiteTestwas x  designedtomeasurethesystemsabilitytoprocessalargevolumeofinformation. ^  TherewasaphilosophicaldifferencebetweentheCensusBureauQAspecialistsandprogram *z  managersregardingtheapplicationofqualitystandardsontheDCS2000component.The ` specialistswereassuredthatqualitymeasurementandcorrectiveactionswereavailable,butit F wasnevercleartothespecialiststhroughtheavailabledocumentationorrepeatedexplanation, , howQAwouldbeappliedandmeasuredduringthedatacaptureprocess.Itwasuncleartothe  programmanagerswhytheirdocumentsandpresentationsdidnotanswertheQAspecialists  questions.    TheDCS2000componentwasatechnologicalsuccess,butwastoosophisticatedforregular Z surveyefforts.Inretrospect,thereuseoftheDCS2000fornoncensusworkmaynotbe @ realisticorideal.Reusecannotbeachieved,unlesstheDecennialCensusitselfbecomesless &v complexorhasfewerspecializedrequirementsthatnecessitatesystemcustomization.  \ Theseandotherfindingsledtothefollowingrecommendations: ( " i "   iQ23  0    Definerequirementsearly.Startingtheplanninganddevelopmentearlierwouldprovide  agreaterchancethatallidentifiedrequirementswouldbeimplementedandthat   sufficienttimewouldexistfortestingandrefinement. iQQ݌p! (#(# Ќ  " i!"   iBS23  0    Takeadvantageofinstitutionalknowledge.Outsourcingshouldprovideameansto <#" augmentandextendthecapabilitiesofinhousepersonnel.Thecorporateknowledgeand "$r# understandingofcensusprocessesneedstobemaintained.Experiencedsystem %X $ developers,operationalcustomers,andcensuscontentexpertsshouldbeactively %>!% involvedthroughouttherequirementsidentificationanddevelopmentprocesses. iBSoS݌&$"&(#(# Ќ  " i""   iU23  0    Developstandardprojectmanagementtoolkit.Theprojectteamdevelopedanumberof (#( processesandtoolstosupporttheircontractmanagementanddevelopmentactivities. )$) Ideally,astandardsetoftoolswouldbeavailableforeachteamprojectpriorto l*%* developmenttoavoiddivertingprojectresourcesfromtheactualcontractmanagement R+&+ andsystemdevelopmenttasks. iU V݌8,',(#(# Ќ  SynthesisofResultsfromtheCensus2000AlternativeQuestionnaireExperiment  (AQE2000) (Martinetal2004)  ThisreportsummarizesthefindingsofthreeexperimentsincludedintheAQE2000.Allthree b experimentswerelimitedtothemailoutmailbackuniverse. H  Theskipinstructionexperiment examinedrespondentperformanceinfollowingskip  d instructionsonthecensuslongform.Itcompareddifferentwaysofaidingrespondents  J navigationthroughthequestionnaire.Onedesignincorporatedinstructionsandvisualfeatures  0 tohelprespondentspreventerrorsbeforetheyoccurredandanotherwasdesignedtohelp    respondentsdetecterrorsaftertheyoccurred.Inadditiontothesepreventionanddetection    designs,otherpotentialdesignimprovements,suchasusingreverseprinttoattractrespondents   attentiontoinstructionsandrewordingthestandardskipinstruction,werealsotested. x  Errorsofcommission(whichoccurwhenrespondentsincorrectlyanswerquestionstheyshould D  haveskipped)weresignificantlyreducedinalloftheexperimentaltreatments,suggestingthat *z  thedesignchangesimprovedrespondentsperceptionandcomprehensionoftheinstruction. ` Errorsofomission(whichoccurwhenrespondentsskipquestionstheyshouldhaveanswered) F decreasedfortheDetectionTreatment,butsignificantlyincreasedforeveryotherexperimental , treatment.Eithertypeoferrorindicatesrespondentdifficultynavigatingthequestionnaire,but  theirimpactisdifferent.Errorsofomissionresultinmissingdata.Errorsofcommission  increaserespondentburdenandfrustration.TherecommendationistoadopttheDetection  methodinmailquestionnaires,sinceitsignificantlyreducesbothtypesoferrors. t  Theresidenceinstructionsexperiment involvedthepresentationofresidencerulesonCensus @ 2000shortform.Thisresearchaimedtoimprovewithinhouseholdcoveragebyrewordingthe &v rosterinstructionstomakethemunderstandabletorespondents,byencouragingrespondentsto  \ readthemthroughappropriateplacementandformatting,andbypresentingtheinstructionsto B increaserespondentswillingnesstofollowthem. ( Thechangesinformat,presentation,andwordingoftheresidenceinstructionsresultedina  significantlyhigherresponsetothehouseholdcountquestion(whichservesasanimportant   indicatorofmissingdataandflagslargehouseholdfollowup).Theexperimentalpanelalso p!  producedsignificantlyfeweromissionsamongHispanicsinthelowcoveragestratum.The V"! recommendationistoconductadditionaltestingofthegraphicalandwordingchangesthatledto <#" theseimprovements,tobetterunderstandtheireffectsandtofurtherimprovethequalityof "$r# householdcountdata. %X $  TheraceandHispanicoriginexperiment comparedthe1990styleraceandHispanicorigin &$"& questionswiththenewquestionsintheCensus2000shortform.Itexaminedtheeffectsof ' #' changesmandatedbytheOfficeofManagementandBudgettoallowthereportingofmorethan (#( oneraceandreversedthesequenceoftheraceandHispanicoriginitems.Otherchangesin )$) format,categories,andwordingwerealsointroducedinCensus2000andtheneteffectsofall l*%* thechangesonraceandHispanicreportingwereanalyzed. R+&+  8,', Overall,thequestionnairerevisionssubstantiallyimprovedthecompletenessofraceand  Hispanicoriginreportinginmailshortformquestionnaires.Inaddition,Hispanicswereless  likelytoreporttheirraceasSomeOtherRace,andmorelikelytoreportasWhite,inthe2000 | stylequestionnaires.Althoughtherewerenoapparentquestionnaireeffectsonthefraction b reportedasHispanic,therewereeffectsonthereportingofdetailedHispanicorigingroups.The H 1990stylequestionnaireobtainedmoredetailedreportsofHispanicoriginthanthe2000style . ~ questionnaire,probablyduetotheeffectsofquestionwordingdifferencesaswellasexamples.  d Unexpectedly,therewerethreetimesasmanyreportsoftheexamplegroupsforAsianand  J PacificIslandergroupsinthe2000styleform,whichdidnotlistexamples,astherewereinthe  0 1990styleform,whereexampleswerelisted.Theexperimentdemonstratesthatsome    questionnairedesignchangesmadeinCensus2000resultedinsubstantialimprovementsindata    quality,butthatotherchangeshadunintendedconsequences.Therecommendationsareto   carefullypretestandfieldtestallchangestothequestionnaireandtoconductsimilarbutlarger x  replicationstudiesinfuturecensusestoevaluatetheeffectsofquestionnairechangesonthe ^  comparabilityofdatafromonecensustothenext. D  Theresultsofallthreeexperimentspointtointeractionsbetweenquestionformatandcontent, ` suggestingthattheCensusBureaumustattendtothecomplexrelationshipsbetweenformatand F meaninginselfadministeredquestionnaires.Thesefactorshavebeendemonstratedheretohave , measurableeffectsonthedata.Theseexperimentsdemonstratethatformataffectsperformance  onbranchinginstructions,affectsresponse/nonresponseonthehouseholdcountquestionand  indirectlyaffectscoverageandthatformatdifferencesbetween1990styleandCensus2000  formsaffectraceandethnicityreporting. t  SynthesisofResultsfromtheAdministrativeRecordsExperimentin2000(AREX2000) &v  (JudsonandBye2004)  \ TheAREX2000wasanexperimentdesignedtogaininformationregardingthefeasibilityof ( conductinganadministrativerecordscensusortheuseofadministrativerecordsinsupportof  conventionaldecennialcensusprocesses.IntheAREX2000,anadministrativerecordcensus  wasdefinedasaprocessthatreliesprimarily,butnotnecessarilyexclusively,onadministrative   recordstoproducethepopulationcountandcontentofthedecennialcensusshortform,witha p!  strongfocusonapportionmentandredistrictingrequirements.Inadditiontototalpopulation V"! countsbystate,thedecennialcensusmustprovidecountsofthevotingage(18andover) <#" populationbyraceandHispanicoriginforsmallgeographicareas,currentlyintheformof "$r# censusblocks. %X $ Demographically,theAREX2000provideddateofbirth,race,Hispanicorigin,andsex. &$"& Geographically,theAREX2000operatedatthelevelofbasicstreetaddressandcorresponding ' #' Censusblockcode.Unitnumbersformultiunitdwellingswereusedincertainaddress (#( matchingoperationsandoneoftheevaluations;butgenerallyhouseholdandfamilycomposition )$) werenotcaptured.ThedesignassumedtheexistenceofaMAFandgeographiccoding l*%* capabilitysimilartothatavailableforCensus2000. R+&+  8,', Twositeswereselectedthatwerebelievedtohaveatotalofapproximatelyonemillionhousing  unitsandapopulationofapproximatelytwomillionpersons.OnesiteincludedBaltimoreCity  andBaltimoreCounty,Maryland.TheothersiteincludedDouglas,ElPaso,andJefferson | Counties,Colorado.Thesitesprovidedamixofpopulationandhousingcharacteristicsneeded b toassessthedifficultiesthatmightariseinconductinganadministrativerecordscensus. H TheAREX2000hadtwoprincipalobjectives:todevelopandcomparetwomethodsfor  d conductinganadministrativerecordscensus,onethatusedonlyadministrativerecordsanda  J secondthataddedsomeconventionalsupporttotheprocesstocompletetheenumerationandto  0 explorethepotentialuseofadministrativerecordsdataforsomenonrespondingorunclassified    householdsthatoccurinaconventionalcensus.    AtwophaseprocessaccomplishedtheAREX2000enumeration.Thefirst,orTopdown,phase x  involvedtheassemblyofrecordsfromanumberofnationaladministrativerecordsystemsand ^  unduplicationofindividualswithinthecombinedsystems.Thiswasfollowedbycomputer D  geocodingofstreetaddressestothelevelofcensusblockandtwoattemptstoobtainandcode *z  physicaladdressesforthosethatwouldnotgeocodebycomputer.Finally,therewasaselection ` of bestdemographiccharacteristicsforeachindividualand beststreetaddresswithinthe F experimentalsites. , Thesecond,Bottomup,phaseoftheAREX2000designwasanattempttocompletethe  administrativerecordsonlyenumerationbythecorrectionoferrorsinadministrativerecords  addressesthroughaddressverification(acoverageimprovementanalogue)andbyadding t personsmissedintheadministrativerecords(aNRFUanalogue).ConsideringtheTopdown Z andBottomupprocessesaspartofoneoveralldesign,theAREX2000canbethoughtofasa @ prototypeforamoreorlessconventionalcensuswiththeinitialmailoutreplacedbyaTopdown &v administrativerecordsenumeration.  \ Therewerefourprincipallimitationsontheexperiment. (  * #$%&'((xx#* #$%&'(" i"   i;23  0    Theadministrativerecordssourcefileswerelimitedtothoseusedinthecreationofthe  StatisticalAdministrativeRecordsSystem(StARS)1999,whichreliedprimarilyonfiles   fortaxyear1998andotherfilesextractedearlyincalendaryear1999.Thesefiles p!  neitherexhaustedthenationalleveladministrativerecordsthatmighthavebeenavailable V"! fortheAREX2000norweretheythemosttimelywithrespecttoApril1,2000,Census <#" DayforCensus2000. i;h݌"$r#(#(# Ќ  " i"   i%23  0    Thenumberofexperimentalsiteswassmall.Althoughitwouldnothavebeen %>!% reasonableorrealistictoattempttomountthisfirstadministrativerecordsexperimentin &$"& arepresentativesampleofgeographicareaslargeenoughtomakenationalestimates, ' #' additionalsiteswouldhaveprovidedmoreconfidencethattheresultswerenot (#( idiosyncratictothesitesselected. i%R݌)$)(#(# Ќ  " i"   i23  0    Therewasnoexperimentalvariationinkeydesignparameterssuchastheclericaland R+&+ fieldoperationsandtheaddressselectionalgorithm.Withoutsomefactorialorfractional 8,', factorialstructure,directestimatesofoperationalimpactsofcomponents,individuallyor  incombination,werenotpossible. i͈݌(#(# Ќ  " i"   iƊ23  0    ThemeasurementofraceandHispanicorigininadministrativerecordsatthenational b levelisdeficient.Attemptsweremadetoimprovethemeasurementthroughtheuseof H certainstatisticalmodels,buttheresultswerenotentirelysatisfactory. iƊ݌. ~(#(# Ќ  Therewerefourevaluations:Process,Outcomes,Household,andRequestforPhysicalAddress  J evaluations.  0  AREX2000ProcessEvaluation     BasicresultsfromtheAREX2000Processevaluationincluded: x  ThereisareportinglagofapproximatelyoneyearbetweentheAREX2000sourcefilesandthe D  targetdateofApril1,2000.Thereportinglagimpactedtheinterpretationofresults. *z  IntheMarylandtestsite,themachinegeocodingratewasapproximately86percent,whileinthe F Coloradositetheratewasapproximately80percent.Theclericalgeocodingprocessadded , about3percenttothenumberofaddressesgeocodedinMarylandandabout5percenttothe  numberofaddressesgeocodedinColorado.  FortheBottomupmethod,administrativerecordaddresseswerecomputermatchedtoanApril t 2000extractoftheDMAF.About80percentofMarylandAREX2000addresseswere Z computermatchedtoatleastoneDMAFaddress,whileabout81percentofColorado @ administrativerecordaddresseswerecomputermatchedtoatleastoneDMAFaddress.A &v clericalreviewofthecomputermatchingprocessaddedanadditional4percentofaddressesin  \ Marylandandnearly6percentofaddressesinColoradobyclericallymatchingaddressestothe B DMAF. ( ForadministrativerecordaddressesthatdidnotmatchaDMAF,fieldaddressverificationwas  performed.Thefieldverificationwasoriginallydesignedfor100percentverification,butdueto   Census2000demand,thefieldverificationwasreducedtoasamplebasiscomposedof6,644 p!  addresses.About13percentoftheMarylandaddresseswerevalidaslisted,whileanadditional V"! 12percentweredeemedvalidafterthelistermademinorcorrections.InColorado,about8 <#" percentwerevalidaslistedandanadditional30percentweredeemedvalidafterminor "$r# correctionsbythelister. %X $ Timeconstraintsdidnotallowforadetailedpersonbypersoncomparisonbetweentheresultsof &$"& theBottomupmethodandtheDecennialCensus,norbetweentheresultsoftheBottomupand ' #' Topdownmethods.AlthoughahouseholdmatchwasconductedbetweentheBottomup (#( methodandthecensus,itremainsanopenquestionwhetherthematchedaddressesinthe )$) Bottomupmethodcontainthesamepeopleasthoseidentifiedinthecensus.Administrative l*%* RecordsResearchshouldperformanevaluationusingadetailedpersonbypersoncomparison R+&+ (micromatch)ofthematchedaddresseswithinthecensusandBottomupmethods. 8,', Additionally,adetailedpersonbypersoncomparisonbetweentheBottomupandTopdown  methodsshouldalsobepursuedwithregardtopersonandaddressmatches.  WhentheAdministrativeRecordExperimentpopulationtallieswereproducedandcomparedto b theCensus2000tallies,theresultsshowedthatfortheBottomupmethod,thefivetestsite H countytallies,rangedfrom96percentto102percentoftheCensus2000populationtallies.For . ~ theTopdownmethod,therangewas8492percent.Basedontheseresults,recommendations  d includethatadministrativerecordscontinuetobetestedandrefinedasapossiblesupplementfor  J futurecensusoperations.Futurerefinementandimprovementsshould,ataminimum,focuson  0 thefollowingareas:     * #$%&'((xx* #$%&'(" i"   i˜23  0    Improvethecomputermatchingandrematchingprocesses. i˜݌ (#(# Ќ  " i"   i23  0    EvaluatetheimpactofmultipleMAFIDsontheDMAF.MultipleMAFIdentifiers ^  assignedtoasingleaddressandduplicateMAFIDsassignedtomultipleaddresses D  contributedtothedifficultyinclassifyingaddressesasmatched,nonmatched,orpossibly *z  matched. i݌`(#(# Ќ  " i"   i23  0    Improvetheavailabilityofsourcedatafortheunder18population. iޟ݌,(#(# Ќ  " i"   i23  0    Evaluatetheeffectivenessofcomputermodelsusedintheexperiment. i٠݌(#(# Ќ  " i"   i23  0    Conductfurtherresearchonaddressselectionrulesusedtodetermineapersons best t address. iա݌Z(#(# Ќ  " i"   iܢ23  0    Conductafullscalefieldaddressverification. iܢ ݌&v(#(# Ќ  AREX2000OutcomesEvaluation B  Asexpected,theBottomupcoverageismuchimprovedcomparedtotheTopdown.Thisis  largelyduetothecompletionoftheTopdownenumerationbyusingcensusdatafor  nonmatchedaddresses,whichsimulatesafollowuptotheadministrativerecordsenumeration.   Specifically,theBottomupcoverageofchildren(8194percentacrossthetestsites)is p!  substantiallybetterthantheTopdown(7283percent).Coverageofchildrenisaparticular V"! weaknessforadministrativerecordsusedintheAREX2000. <#" AdultsintheBottomuparemoreorlessuniformlyovercounted(102104percent).The %X $ overcountofadultsmostlikelyisduetounrecordeddeathsthatoccurredinthe12monthsprior %>!% toCensusDay,thelackofspecialpopulationsoperationsintheAREX2000(e.g.,agroup &$"& quartersenumeration),andfailuretounduplicatepersonsafteraddingcensusdatafor ' #' nonmatchedaddresses.Thelattermeansthatthereissomeduplicationofchildrenaswell. (#( DetailedenumerationresultsfocusedmainlyonacomparisonoftheBottomupenumeration l*%* withtheCensus2000.Theanalysisdidnotincludegroupquartersand,duetolimitationsinthe R+&+ administrativerecordssources,personscouldnotbereportedwith multior otherrace.The 8,', analysisprogressedfromlargegeographicareastosmallgeographicareas,beginningwiththe  fivetestsitecountiesandendingwithcensusblockswithinthesites.Theevaluation  incorporatedavarietyofmethodstoaccomplishitsobjectives,includingunivariateand | multivariatestatisticalanalysesoftheAREX2000/Census2000differences,and b spatial/ecologicalmapsthatexaminedthegeographicdistributionsofkeycomparisonmeasures. H TheoutcomesevaluationtriedtodisentangletheinfluenceofdemographicchangeandAREX . ~ 2000processing,coverage,anddataqualityissues,whilepresentingbasicenumerationstatistics.  d Atthecountylevel,theBottomupprocessundercountedtotalpopulationinallsitesexcept  0 BaltimoreCity.Aswiththetotalpopulation,malesandfemaleswereundercountedinallsites    exceptBaltimoreCity,butthefemaleundercountswereslightlygreaterthanmaleundercounts.    Agegroupsshowedmorevariabilitywithmostgroupsundercounted.Generallythesizeofthe   undercountsincreasedwithdecreasingage,exceptforthe2024agegroup.Thesepatternsdid x  notappeartobesitespecific.Overcountsfortheoldestoldandundercountsfortheyoungest ^  personssuggestthatmuchmoretimelybirthanddeathinformationmustbeobtained.Also,the D  specialenumerationrequirementsforpopulationssuchascollegestudents,themilitary,and *z  personsinnursinghomesmustbeincorporatedintoadministrativerecordsprocesses. ` AdministrativerecordsarenotcurrentlyagoodsourceofdataforraceandHispanicoriginand , themodelswerenotsufficienttocorrecttheirdeficiencies.BlacksandHispanicswere  undercountedwhentheywerealargeminoritygroupandovercountedwhentheywerenot.  AmericanIndiansandAlaskannativeswerenotwellidentifiedandtheaccuracyofAsian/Pacific  Islandercountswasuncertain. t Bottomuptractleveltotalpopulationresultsindicatedagoodcorrespondencebetweenthe @ AREX2000andCensus2000.Thepopulationcountsof70percentoftractswerewithin5 &v percentagepointsand95percentofthetractswerewithin25percentagepoints,thoughasizable  \ numberoftractshadmoderateandlargeundercounts.Attheblocklevel,populationcounts B weretheleastaccurate.Forthetotalpopulation,38percentofblocksmetthe5percentcriterion ( andabout85percentofblocksmetthe25percentcriterion.  Amultivariateanalysisofblockdifferencesshowedthatlargeundercountswereassociatedwith   suchblockcharacteristicsashighpopulationdensity,highrentalrates,andlargeproportionsof p!  personsage2024.Largeovercountswereassociatedwithhighvacancyrates,lowpopulation V"! density,smallproportionsofpersonsundertheageof20,andlargeproportionsofpersonsage <#" 2024andage65andover. "$r#  AREX2000HouseholdlevelAnalysis  %>!% Thegeneralgoalofthehouseholdlevelanalysiswastoassesshowwellhouseholdsformedfrom ' #' administrativerecordsmatchedthosefromCensus2000addresses.Theevaluationfocusedfirst, (#( onthefactorsassociatedwithAREX2000andCensus2000addressesthatwere(computer) )$) linked.Then,demographiccomparisonsweremadebetweenhouseholdsatlinkedaddresses. l*%*   8,', TherewasaspecialfocusonCensus2000householdsthatrequiredaNRFUvisitandCensus  2000unclassified(imputed)households.  Theevaluationusedbothdescriptiveanalysesandlogisticregressionanalysistoassessthe b coverageandaccuracyofAREX2000households.Descriptiveanalyseswereperformedfor H householdsinallfiveAREX2000countiesandfortheCensus2000NRFUandimputed . ~ householdsinthetestsites.Alogisticregressionmodelwasdevelopedtopredicttheprobability  d ofanaccuratehouseholdmatchusingaddressandAREX2000processingcharacteristicsas  J predictors.Addresseswithahighprobabilityofcorrectdemographicmatchbetweenoccupants  0 mightbecandidatesforadministrativerecordssubstitutioninthecaseofNRFUina    conventionalcensus.Inthefollowingdiscussiontheterm linkedisusedtomeanamatched    address.Theterm matchedisreservedforhouseholddemographiccomparisonsatlinked   addresses. x  AREX2000scoverageofthecensusNRFUuniversewasnotasgoodasitscoverageofthe D  overalluniverse.AREX2000housingunitswerelinkedwith70.9percentofthecensusNRFU *z  housingunits,comparedwith88.4percentofthecensusrespondinghousingunits.Foroccupied ` NRFUhousingunits,thecoverageratewas76.7percent.TheAREX2000housingunitswere F linkedwith63.2percentofhouseholdsthatwereimputedtohavepeopleinthemand34.7 , percentofthoseimputedtobevacant.  TheAREX2000andthecensuscountedthesamenumberofpeopleinthehousingunitfor51.1  percentofthe889,638linkedhouseholdsandAREX2000waswithinoneofthecensusfor79.4 t percentoftheunits.The51.1percentiseffectivelyaceilingonthepercentoflinkedhouseholds Z thathadexactlythesamepersonsfromAREX2000andCensus2000.Althougherrorsin @ addresslinkagewouldaccountforsomeofthemismatchedhouseholds,thedeficienciesin &v administrativerecordscitedearlierinthisreport(missingchildren,lackofspecialpopulation  \ operations,andthetimegapbetweentheadministrativerecordsextractsandCensusDay)most B likelyaccountforthemajorpart. ( ForlinkedNRFUhousingunits,AREX2000hadthesamenumbersofpersonsfor37.0percent  oftheunitsandwaswithinone69.3percentofthetime.Census2000NRFUhousingunitswere   moresusceptibletotheAREX2000deficienciesthanrespondingunits.Inaddition, p!  enumerationerrorsinCensus2000mighthavebeenhigherfortheseunits. V"! Theregressionanalysisdemonstratedanumberoffactorsassociatedwithgreaterprobabilityof "$r# matchedhouseholddemographics.Theseincludesingleunitaddressratherthanmultiunit, %X $ householdwithonlyoneortwomembers,allhouseholdoccupantsovertheageof65,atleast %>!% oneWhiteoccupant,andnooccupantwithimputedraceintheAREX2000.Thepredictive &$"& powerofthemodelwasmoderatelystrong.Atapredictedprobabilityof0.5orhigher,the ' #' probabilityofacorrecthouseholdmatchwasabout72percent.Atapredictedprobabilityof0.8 (#( orhigher,theprobabilityofacorrectmatchincreasedtoabout83percent,buttheproportionof )$) addresseswithpredictedprobabilitythishighwasonlyabout4percentofalladdresses. l*%* Evidently,thelimitationsinthedata,particularlytheadministrativerecordscutoffsandpoor R+&+ raceandHispanicoriginmeasurement,madehouseholdpredictionquitedifficult. 8,', Ї AREX2000RequestforPhysicalAddressEvaluation   TheAREX2000RequestforPhysicalAddressOperationcollectedphysicaladdresses | (geocodablehousenumber,streetname,cityandstate)forindividualswhoseaddresswaslisted b asaP.O.Boxorothernoncitystyleaddressfromsixadministrativerecordssourcefiles.Major H componentsoftheoperationweretocreateanaddressfilefromadministrativerecordswherethe . ~ mailingaddresswasaP.O.Boxorothernoncitystyleaddress,designaformandmailittothe  d addresses,requestingaphysicaladdress,clericallygeocodethephysicaladdressestostate,  J countyandblock,andkeyaddressesandgeocodeinformationtoafileforfurtheranalysis.  0 Themailingrequestingaphysicaladdressincluded138,653individualsintheadministrative    recordsfiles.Attheconclusionoftheoperation,9,431physicaladdressesweregeocoded,of   which8,107weregeocodedtoatestsitecounty.Whiletheinitialplanfortheoperationcalled x  forincorporatingthegeocodedaddressesintotheAdministrativeRecordExperimentfiles,the ^  lowreturnrate,combinedwithresourcelimitations,ledtoadecisiontonotupdatethefileswith D  theinformation.Instead,theresultswerecontainedinaseparateevaluationreport. *z  Whererespondentsreturnedphysicaladdressinformation,addresseswereabletobegeocodedto F thestate,county,andblocklevelandweredefinedasbeinginoroutofthetestsite.Thetiming , ofthemailingcontributedtoalowreturnrate.  Unintendedrecipientsaffectedtheresults.Anunderlyingphenomenonregardingthemailingis  thatthepersonwhoreceivedthemailingandrespondedmaynotbethepersonthatwaslinkedto t thatP.O.boxintheadministrativerecordssourcefiles.Thiscanbeattributedtothetime Z elapsedbetweenthedateoftheadministrativerecordsourcefiledataandthemailingofthe @ RequestforPhysicalAddressmaterialsandthetypicallyhighturnoverrentalrateofP.O.Boxes. &v Thedesignoftheformimpactedtheresults.Althoughtheformgenerallysuitedthepurposeof B theRequestforPhysicalAddressoperation,commentsreturnedontheformsuggestsomeareas ( oftheformcouldhavebeenexpandedorreviewedtoimprovethequalityofinformationthat  wasreturned.  BasedonananalysisoftheresultsoftheRequestforPhysicalAddressOperation, p!  recommendationsincludethefollowingactions: V"! " i"   i23  0    Assesstheimpactofformdesignandtimingofthemailinginaremailingtotheoriginal "$r# addressees. i݌%X $(#(# Ќ  " i"   i 23  0    Assesstheimpactoftheformdesignandtimingofthemailingbyusinganotheruniverse &$"& ofadministrativerecordaddresses. i 8݌' #'(#(# Ќ  " i"   iR23  0    Assesstheneedforthistypeofoperationanddetermineifcollectingphysicaladdresses )$) onindividualswhohavenootheraddresstypeinadministrativerecordsourcefilesmerit l*%* thecostoftimeandresourcestolaunchaseparateoperationtogathertheseaddresses. iR݌R+&+(#(# Ќ   8,', OverallAdministrativeRecordsimplicationsfor2010Censusplanninginclude:  ContinuetoexplorethepossibilityofusingadministrativerecordsasasubstitutionsforNRFU | inthe2010Census.Althoughtheresultsofthehouseholdlevelanalysiswerenotdefinitivedue b tothelimitationsonAREX2000,theyweresufficientlystrongthatresearchintothesubstitution H ofadministrativerecordshouseholdsforNRFUorunclassifiedhouseholdsinaconventional . ~ censusshouldcontinue.ForNRFUhouseholdsthereisthepotentialforsignificantcostsavings,  d andforunclassifiedhouseholds,thepotentialforgreateraccuracythanthatprovidedby  J imputation.  0 Thereareotheraspectsof2010Censusdevelopmentinwhichadministrativerecordsmightplay    arole.TheseincludeMAFimprovements,developmentandtestingofunduplicationmethods   for2010,subnationalDemographicAnalysis,andcoveragemeasurementresearch. x  Arrangementsshouldbemadetoacquireadministrativerecordsonatimelierbasisandtoobtain D  somedatasetsthatmightfillsomeoftheadministrativerecordscoveragegaps. *z  Aresearchagendafor2010couldinclude: F " i "   i23  0    AdditionalevaluationoftheimpactofclericalandfieldoperationsinAREX2000. i݌(#(# Ќ  " i "   i23  0    PersonunduplicationintheAdministrativeRecordsExperimentBottomupprocess. i݌(#(# Ќ  " i "   i23  0    RepeatingAREX2000withStARS2000data. i%݌Z(#(# Ќ  " i "   i23  0    RepeatingtheHouseholdlevelanalysisusingStARS2000data. i݌&v(#(# Ќ  " i "   i23  0    Analysisofadministrativerecordscoveragegaps,inparticulargapsrelatedtopersonsin B groupquarters. i݌((#(# Ќ  " i"   i23  0    MAFimprovementsusingadministrativerecords. i.݌(#(# Ќ  " i"   i23  0    Improvingaddresslinkagetechniques. i݌p! (#(# Ќ  " i"   i23  0    EnhancingNumidentraceandHispanicorigindatausingCensus2000. i݌<#"(#(# Ќ  " i"   i23  0    ContributingtosubnationalDemographicAnalysis. i݌%X $(#(# Ќ   ' #'  SynthesisofResultsfromtheSocialSecurityNumber,PrivacyAttitudes,andNotification  ExperimentinCensus2000(SPAN) (LarwoodandTrentham2004)  TheSPANExperimentwasdesignedtoassessthepublicsattitudesonprivacyand b confidentialityissuesrelatedtothenotionofanadministrativerecordscensusandtofurther H examinehowthenotificationofadministrativerecordsuseandtherequestforaSocialSecurity . ~ NumberwouldimpactcensusresponseratesanditemnonresponseratesduringCensus2000.  d Theprojectincludedasurveyandapanelcomponent,enablingbothattitudinalandbehavioral  J responsestobeevaluated.  0 Thesurveycomponent(StudyofPrivacyAttitudesin2000)wasconductedbytheInstituteof    SocialResearchattheUniversityofMichiganandTheGallupOrganization.Thiscomponent   gatheredinformationonpublicattitudesregardingthecensus,itsuses,trustandprivacyissues, x  theCensusBureausconfidentialitypractices,possibledatasharingacrossFederalagencies,and ^  thewillingnesstoprovideonesSocialSecurityNumber.Telephonesurveyswereconducted D  withtwodifferentsamplesofU.S.householdresidentsin1999and2000,beforeandafter *z  CensusDay2000.Themajoranalysesincluded(a)comparisonsoftheresponsestothoseof ` similar1995and1996publicsurveyscommissionedbytheCensusBureautoassesslongterm F attitudinaltrends,(b)comparisonsbetween1999and2000responsestoexamineanypotential , effectsthecensusenvironmentmayhavehaduponpublicattitudes,andc)theassessmentof  howselfreportedcensusmediaexposurebyCensus2000surveyrespondentsmayhave  impactedtheirresponses.Respondentsaddresseswerealsoobtainedtoexaminehowpredictive  respondentsattitudeswereoftheirbehaviorofactuallyreturningtheCensus2000form. t Relationshipsbetweenrespondentsattitudes,demographicinformation,exposuretocensus Z publicity,andresponsebehaviorweresubsequentlydetermined. @ Thepanelcomponentconsistedoftwostudiesexaminingrespondentsbehavioralresponsesto  \ actualSocialSecurityNumberrequestsand/orpublicnotificationofadministrativerecorduse. B TheSocialSecurityNumberNotificationstudyevaluatedtheeffectsoftheSocialSecurity ( Numberrequestandthenotificationofadministrativerecordsuseuponmailresponseratesand  formcompleteness.TheSocialSecurityNumberValidationstudyfocusedupontheaccuracyof  SocialSecurityNumbersprovidedbyrespondentsandexaminedtheeffectoftherequestand   administrativerecordsnotificationupontheirvalidationrates.Bothstudiesuseddatacollected p!  duringCensus2000.Tenpanelsweredesignedwithdifferentexperimentaltreatments.The V"! experimentalcoverlettersandformsweretheofficialcensusformsreceivedbythesampled <#" households,inthestandardsequenceandtiming.AllSocialSecurityNumberrequestswere "$r# voluntary. %X $ Inbrief,theresultsoftheSurveyofPrivacyAttitudesin2000indicatedthat: &$"&  * #$%&'((xx* #$%&'(" i"   i23  0    Thepublichassteadilyincreaseditsknowledgeandawarenessofthecensus,itsuses, (#( andlawsrelatedtoconfidentialitypracticesbetween1995and2000.TheCensus2000 )$) publicityseemedtoenhancethepublicsknowledgeandendorsementtocooperatewith l*%* thecensus. i ݌R+&+(#(# Ќ   8,', " i"   i23  0    LongtermsurveytrendsshowedincreasesinthepublicsbeliefthattheCensusBureau  actuallyprotectsdataconfidentiality;howevernochangeswereshowninthepublics  trustintheCensusBureautokeepdataconfidentialbetween1999and2000,suggesting | noeffectbythecensuspublicityuponpublicattitudesrelatedtoconfidentialityissues. i!݌b(#(# Ќ  " i"   iK23  0    Generalprivacyconcernsshowedaverysmall,yetstatisticallysignificant,decline . ~ between1999and2000;howeverlongtermtrendsshowsmallincreasesinpublic  d concernsaboutpersonalprivacyandthelossofcontroloverpersonalinformation.The  J proportionwhoviewedthecensusasaninvasionofprivacydidnotchangebetween1999  0 and2000. iKx݌  (#(# Ќ  " i"   i23  0    Trendsrevealedthatincreasingpercentagesexpressdisapprovaltowardsdatasharingor   providingonesSocialSecurityNumber.Around45percentin1999and2000stated x  thatitwouldbotherthem alotiftheircensusinformationwasshared,asignificant ^  increasefromprioryears.ExpressedwillingnesstoprovideonesSocialSecurity D  Numberdeclinedfrom68percentin1996to55percentin1999,withnochangein2000. i݌*z (#(# Ќ  " i"   in23  0    RelationshipswererevealedbetweenCensus2000surveyrespondentsattitudesandself F reportedexposuretocensusrelatedmedia.Thoseexposedtobothpositiveandnegative , mediaweremoreknowledgeableaboutthecensus,considereditmoreimportant,and  weremorelikelytoendorseanobligationtocooperatewiththecensusthanthosewithno  mediaexposure.Theonlynegativeexposuregrouphadsimilarresponsestothosewith  bothpositiveandnegativemediaexposure,whilemoredifferenceswereshownbetween t theonlypositiveexposuregroupandthosewhoreportedexposuretobothtypesof Z censusrelatedmedia. in݌@(#(# Ќ  " i"   i#23  0    Attitudeswereshowntopredictrespondentsbehavior,withhighprivacyconcerns,  \ negativeviewsontheCensusBureausconfidentialitypractices,disapprovalofdata B sharing,andalackofwillingnesstoprovideSocialSecurityNumbers,beingreliable ( negativepredictorsofwhetherrespondentsreturnedtheirCensus2000formsand  providedmailingaddressesthatcouldbeusedtodeterminethereturnstatusoftheir  forms.Usingreporteddemographics,nonWhiterespondentswereshowntobeless   likelytoreturntheirforms. i#P݌p! (#(# Ќ  TheSocialSecurityNumberNotificationpanelstudyresultsrevealedthat: <#" " i"   i23  0    TheSocialSecurityNumberrequestforoneorallhouseholdmembersdecreasedmail %X $ responserates,yetthedecreasesweresmallerthanexpected.Specifically,results %>!% suggestedthattheSocialSecurityNumberrequestforallhouseholdmemberswould &$"& decreaseresponseby2.1percentinhighcensuscoverageareasand2.7percentinlow ' #' censuscoverageareascomparedtonorequest.Thedifferencebetweenthedropin (#( responseratesofthehighandlowcoverageareaswasnotstatisticallysignificant. i݌)$)(#(# Ќ  " i"   i 23  0    TheSocialSecurityNumberrequestforallhouseholdmemberswasassociatedwith R+&+ moremissingdata,yettherewasnoeffectshownforPerson1. i  ݌8,',(#(# Ќ  " i"   i7 23  0    Takentogether,specificandgeneralnotificationofadministrativerecordusewasshown  todecreasemailresponse.Separately,however,specificnotificationdidnotdemonstrate  thepredictedstrongereffectsthanthegeneralnotification.Furthermore,therewasnot | sufficientevidencetoconcludethatnotificationfurtherdiscouragedresponseinthe b presenceofaSocialSecurityNumberrequestcomparedtonotificationalone. i7 d ݌H(#(# Ќ  " i "   i23  0    Notificationwasnotshowntoaffectitemnonresponserates,whetherthetwonotification  d typesweregroupedtogetherorexaminedseparately.Further,therewerelowerresponses  J totheSocialSecurityNumberitemforPerson1whentherequestwasmadewithout  0 notification(contrarytoprediction).Thisoccurredregardlessofwhosenumberswere    requested(Person1onlyversusallhouseholdmembers)andregardlessofthe    notificationtype.Also,therewerenoindividualeffectsuponformcompletenessbytype   ofnotification. i݌x (#(# Ќ  Finally,theSocialSecurityNumberValidationpanelstudyresultsshowedthat: D  " i "   i23  0    TherewasahighdegreeofaccuracyfortheprovidedSocialSecurityNumbers,withan ` overallmatchrateof94.8percentbetweentheprovidednumbersandCensusNumident F file(providedbytheSocialSecurityAdministration).Only5.2percentofthereported , SocialSecurityNumberswereconsideredinvalid. i݌(#(# Ќ  " i "   i23  0    ThevalidSocialSecurityNumberratesforhighandlowcoverageareasrevealedasmall,  butstatisticallysignificant,2.4percentdifferencebetweentheaccuracyratesof t respondentsreportednumberswithinthetwocoverageareas(high,95.2percent,and Z low,92.8percent). i݌@(#(# Ќ  " i "   i23  0    ThevalidSocialSecurityNumberratesforPerson1werenotaffectedbywhethera  \ SocialSecurityNumberrequestwasmadeforPerson1onlyorallhouseholdmembers. B Person1validrateswerehighacrossthepanels(about9697percent).Resultsalso ( revealedpatternsofdecreasingvalidationratesforPerson2,Person3,andsoonthrough  Person6amongthepanelsthatrequestednumbersforallhouseholdmembers.  Nevertheless,theirvalidrates,werehighwitharangeofover95percenttothelowest   rateof80.2percentforPerson6. i݌p! (#(# Ќ  " i "   i23  0    Notificationofadministrativerecordsusehadnoeffectuponthevalidationratesof <#" providedSocialSecurityNumbersforPerson1.Also,therewerenodifferencesbetween "$r# thevalidratesofthosewhoreceivedthespecificnotificationtypeversusthegeneral %X $ notificationtype. iD݌%>!%(#(# Ќ  Baseduponthefindingsofthethreestudies,thefollowingrecommendationsweremade: ' #' " i"   i23  0    Designresearchthatfurtherexplorespublicattitudesonprivacy,confidentiality,and )$) trustintheCensusBureau,andtestsmoreeffectivewaystoaddresstheseissuesinfuture l*%* publicityefforts. i݌R+&+(#(# Ќ   8,', " i"   iI23  0    AssessthepotentialimpactofSeptember11,2001(andtheextrasecurityconcernsthat  followed)uponpublicattitudes. iIv݌(#(# Ќ  " i"   i 23  0    Conductqualitativeresearchwithmembersoftargetedpopulationsegmentsthatshow b lowermailresponserates,lessacceptanceofdatasharing,andlesswillingnessto H provideSocialSecurityNumbers,tobetterunderstandtheirperspectivesand . ~ reservations. i  ݌ d(#(# Ќ  " i"   i"23  0    Designresearchtoidentifyother,currentlyunknownbarrierstocensusresponses  0 (besidestheattitudesanddemographicsidentifiedinthepresentanalysis)toultimately    reducethem. i""݌  (#(# Ќ  " i"   i$23  0    ConductmoreresearchontheeffectofSocialSecurityNumberrequestsuponresponse x  behaviorthatfurtherexamines:thecharacteristicsofhouseholdsthatprovideanddonot ^  providenumbers,theaccuracyofhouseholdsreconstructedfromadministrativerecords, D  andtheeffectofhavingandnothavingthenumberinhouseholdreconstruction. i$G$݌*z (#(# Ќ  " i"   ib&23  0    PerformresearchthatfocusesupontheSocialSecurityNumberrequestsofallhousehold F memberstoidentifyfactorsotherthanattitudes(e.g.,practicalbarriers),thatmay , contributetothenonresponseratesofSocialSecurityNumberrequests,anddevelopnew  techniquesthatmayovercomethesenonattitudinalfactors. ib&&݌(#(# Ќ  " i"   i(23  0    Designresearchtofurtherexaminetheeffectofgeneralandspecificnotificationupon t responsebehaviorbyconsideringotherinterpretationsofhowtheymaybeviewed(e.g., Z justifications),andbydevelopingnewmethodsthatfurtherestablishtherelationship @ betweennotificationtreatmentconditionsandbehavior.Futureresearchalsoneedsto &v assessifprovidinginformationontheuseofSocialSecurityNumbersdoesnotmarkedly  \ decreaseresponseratesandimprovesvalidationrates,asthismaychangefuture B censuses. i((݌((#(# Ќ  " i"   i+23  0    DevelopresearchtoassessthecumulativenonresponsetoSocialSecurityNumber  requests(i.e.,unitnonresponse,itemnonresponse,andinvalidrates)toobtainan   indicatoroftheextenttowhichmatchingtoadministrativerecordscouldtakeplace. i+,݌p! (#(# Ќ  " i"   i-23  0    Conductacost/benefitanalysisthatfullyassessesallimplications,shouldtheCensus <#" BureauconsideraskingcensusrespondentsforSocialSecurityNumbersinfuture "$r# decennialcensuses.Futureresearchcouldalsodocumenttheuseofotheridentifiersthat %X $ areusedtolinkfileswithfewercosts. i--݌%>!%(#(# Ќ   (#(  SynthesisofResultsfromtheResponseModeandIncentiveExperimentin2000(RMIE)   (Caspar2004)  TheRMIEinvestigatedtheimpactofthreecomputerassisteddatacollectiontechniquesonthe b responserateanddataqualityinCensus2000.ThethreetechniqueswereCATI,Internet,and H IVR.Householdsparticipatinginthestudywererandomlyassignedtosixpanelsandtoa . ~ controlgroup.ThehouseholdsinthesixpanelsweregiventhechoiceofprovidingtheirCensus  d 2000dataviatheusualpaperformsorbyoneofthealternatecomputermediatedresponse  J modes.Halfofthepanelswereofferedanincentive,atelephonecallingcardgoodfor30  0 minutesofcalls,forusingthealternateresponsemode.    Inaddition,theexperimentincludedanonresponsecomponentdesignedtoassesstheeffectsofa   promisedincentiveandalternativeresponsemodeoptionsonresponseamongasampleof x  censushouseholdswhofailedtoreturntheircensusformsbyApril26,2000.Theintentofthe ^  nonresponsecomponentwasnottotestincentivesorresponsemodeoptionsaspossible D  nonresponseconversiontechniquesforthecensus.Rather,theexperimentwasdesignedtotest *z  theeffectofthesefactorsonresponseamongagrouprepresentingthosewhoaretraditionally ` difficulttoenumerate. F AfinalcomponentoftheexperimentinvolvedinterviewinghouseholdsassignedtotheInternet  mode(bothwithandwithouttheincentive)whooptedtocompletethetraditionalpapercensus  formtodeterminewhythesehouseholdsdidnotusetheInternet.  ResultsfromtheinitialmailoutportionoftheRMIEshowed: Z XzXXXz  7  78݌̌XzXXXz   * #$%&'((xx* #$%&'(" i"   i823  0    CATIbroughtaboutasmallbutstatisticallysignificantimprovementintheoverall &v responserate.Italsohadalowitemnonresponserate. i89݌ \(#(# Ќ  " i"   iA:23  0    TheInternetmodeyieldedrelativelyhighdataquality.Thebenefitsofthisdata ( collectionmethodmayoutweighthecosts. iA:n:݌(#(# Ќ  " i"   i;23  0    TheimplicationsofthisstudyarecomplexfortheuseoftheIVRtechnology.Data   qualitywasthelowestforthismode.Respondentsappearedtodislikelengthysurveys p!  withthismethodandsomerespondentsubgroups(mixedracerespondentsand V"! Hispanics)weremorelikelytoreportconfusionwiththetask. i;;݌<#"(#(# Ќ  " i"   i=23  0    Thecallingcardwasveryeffectiveinpromotingtheuseofthealternativeresponse %X $ mode.However,ratherthanencouragingmorehouseholdstoparticipate,theincentive %>!% tendedtoredirecthouseholdsthatwouldhaverespondedbymailtothealternate &$"& computermediatedresponsemode.Thiseffectmaybepartiallyattributabletothe ' #' colorfulinsertsinthehouseholdmailingthatdirectedattentiontothecallingcard. i==݌(#((#(# Ќ   8,', ResultsfromthenonresponsecomponentoftheRMIEshowedthat:   * #$%&'((xx* #$%&'(" i"   iIA23  0    CATIelicitedthehighestresponsefromCensusnonrespondents(7.8percent)followed | bytheIVRQuestionnaire(4.8percent)andtheInternet(3.7percent).Thiscomparisonis b confoundedbythefactthatInternetaccessmaybeespeciallyproblematicforthistarget H population. iIAvA݌. ~(#(# Ќ  " i"   iSC23  0    RespondentstotheIVRmodearesignificantlyyoungerandresideinhouseholdswith,on  J average,fewerpeoplethanbothmailandComputerAssistedTelephoneInterview  0 respondents.ComputerAssistedTelephoneInterviewrespondentsaredisproportionately    Black,withmorehouseholdsresidinginlowcoverageareascomparedtoInternet    respondents. iSCC݌ (#(# Ќ  " i"   iE23  0    Thecallingcardincentiveincreasedresponsetothealternativemodesby1.9percent ^  acrossallresponsemodes. iEE݌D (#(# Ќ  " i"   iF23  0    Person1inhouseholdsreceivingtheincentiveduetoalternativeresponsemode ` participationtendedtobeyoungerthanPerson1inhouseholdsnotreceivingthe F incentive. iFG݌,(#(# Ќ  " i"   i|H23  0    Contrarytopastresearch,theincreaseinresponseduetotheincentiveisnotstatistically  differentinareaswithhighconcentrationsofBlackandHispanicpopulationsandrenters  (1.9percent)fromotherareas(2.0percent). i|HH݌t(#(# Ќ  " i"   iBJ23  0    Whentotalresponsetoanexperimentalsecondmailingisconsidered,nosignificant @ incentiveeffectremains.Thatis,whenmailresponsesareincludedasrespondents,the &v incentivegroup(13.8percent)isnomorelikelytorespondthanthenonincentivegroup  \ (13.2percent).Similartotheinitialmailoutexperiment,itappearsthattheincentive B merelyredirectsresponsesthatwouldhaveotherwisebeenobtainedbymailto ( alternativemodes. iBJoJ݌(#(# Ќ  " i"   iM23  0    Irrespectiveoftheexperimentaltreatments,around13percentparticipationwasobtained   fromcasesthatdidnotinitiallyreturnthequestionnaireorreturnedthequestionnaire p!  late.Replacementquestionnaireswerenotincludedinthesecondmailing,implyingthat V"! respondentswhoreturnedamailformusedtheiroriginalquestionnaire. iMKM݌<#"(#(# Ќ   * #$%&'((xx* #$%&'(i8Finally,resultsfromtheInternetUsageSurveyindicatedthat: %X $  #$%&'((ۈxx* #$%&'(" i"   iP2ۈ"3  0    Approximately63percentoftheInternetUsageSurveysamplereportedhavingaccessto &$"& theInternet.Thus,accessdoesnotappeartobeamajorreasonwhythesecensus ' #' respondentsdidnotopttocompletetheircensusformviatheInternet. iPP݌(#((#(# Ќ  " i"   iR2ۈ"3  0    NearlyhalfoftheInternetUsageSurveyrespondentsreportedtheywereunawarethatthe l*%* Internetwasanoptionforcompletingtheircensusforms. iRR݌R+&+(#(# Ќ   8,', " i"   iT2ۈ"3  0    AmongrespondentswhowereawareoftheInternetoption,35percentreportedthatthey  believedthepapercensusformwouldbeeasiertocomplete.Otherreasonsfornotusing  theInternetincluded:noaccesstoacomputer,concernsaboutprivacy,forgottheInternet | wasanoption,andinsufficientknowledgeoftheInternet. iT.T݌b(#(# Ќ  " i"   i:V2ۈ"3  0    RespondentsreportedthatanincentivetocompletethecensusviatheInternetwould . ~ haveencouragedthemtousethisalternativemode.About41percentofrespondents  d whowerenotofferedtheincentiveorwereunawareoftheoffersaidtheywouldfillout  J theircensusformviatheInternetiftheywereoffereda30minutecallingcard.Another  0 9percentindicatedtheywoulddoitfora60minutecallingcard,andanadditional12    percentwouldbewillingifa90minutecallingcardwasoffered. i:VgV݌  (#(# Ќ  BasedontheRMIE,thefollowingweretheoverallresults: x   #$%&'((#Kxx #$%&'(" i"   iZ2#K3  0    TheInternetisanattractivealternativedatacollectionmodeforthedecennialcensus. iZIZ݌D (#(# Ќ  " i"   i+[2#K3  0    Theuseofanincentivewasaneffectivemeansofpromotingtheuseofthealternative ` responsemodes.However,someofthiseffectmaybeattributabletotheuseoftheinsert F whichdrewtherespondentsattentiontotheavailabilityofthealternativemode. i+[X[݌,(#(# Ќ  " i"   i]2#K3  0    DataqualitywasimprovedfortheCATImode(ascomparedwithmail).However,this  modeentailssubstantialcostinvestmentsforhardware,software,andprogrammerand  interviewertime. i]>]݌t(#(# Ќ  " i"   i^2#K3  0    Withoutsignificantimprovementsinthevoiceuserinterface,theIVRtechnologyis @ probablynotaviablealternativeforCensus2010. i^^݌&v(#(# Ќ  " i"   i_2#K3  0    Theuseofalternativeresponsemodesdidnotincreaseoverallresponseratestothe B census. i_(`݌((#(# Ќ  ResultsfromtheRMIEsuggestseveralareasworthyoffutureresearch:   * #$%&'((xx #$%&'(" i"   ib23  0    Researchisneededtodeterminethebestwaystopresenttheresponsemodealternatives, p!  asitappearsthatsomerespondentsassignedtothenoincentivetreatmentsdidnotread V"! theletterthataccompaniedtheirpapercensusforminformingthemofthealternative <#" modeoption.Theuseofacolorfulmailinginsert,irrespectiveofwhetheranincentiveis "$r# offeredmaybeenoughtoattractrespondentstoanalternativecensusmode.However, %X $ thisinformationcannotbedeterminedfromthedataobtainedfromthisexperiment. ib-b݌%>!%(#(# Ќ  " i"   i'e23  0    Researchisneededtodeterminewhetherrecentadvancesinspeechrecognitionsoftware ' #' canimprovethevoiceuserinterfacetoincreasedataqualityandeliminatesomeofthe (#( dissatisfactionvoicedbyrespondentswhoansweredtheIVRQuestionnairesatisfaction )$) survey. i'eTe݌l*%*(#(# Ќ  _ 8,', " i"   iMg23  0    Thechoiceofincentiveshouldberevisited.Basedonthenumberofrespondentswho  neverusedtheircallingcardoncetheywereactivated,itappearsthatthecardmaynot  havebeenapowerfulincentive. iMgzg݌|(#(# Ќ  ADemonstrationoftheOperationalFeasibilityoftheAmericanCommunitySurvey  . ~ (GriffinandObenski2001)  d TheACSisindispensabletothesuccessfulreengineeringofthe2010Censusdesign.Tomeet  0 thechallengesofrapiddemographicandtechnologicalchangeandinresponsetostakeholders    requests,CensusBureaumanagershaveconcludedthatthedesignofthedecennialcensusmust    besimplifiedandlongformdatacollectionmustbemoretimely.Therefore,ratherthan   occurringaspartofthedecennialcensus,collectionofdemographicandsocioeconomicdata x  willbeongoingthroughoutthedecadeviatheACS. ^  TheformalprogramtodevelopandtesttheACSbeganin1994.Sincethen,theprograms *z  name,size,andscopehavebeenevolvinginpreparationforfullimplementationin2003.The ` fullsetoftestinganddevelopmentalactivitiesstartingin1994arereferredtoastheACS F developmentprograminthisreport.Keyresultswillbedocumentedinaseriesofreports. , Thisinitialreportfocusesonthedesirabilityandoperationalfeasibilityoffullimplementationof  theACS.In2000,ACSmethodswereemployedin36counties.Anationwidesurvey!the  Census2000SupplementarySurvey(C2SS)!wasconductedinanadditional1,203counties. t TheprimarypurposeoftheC2SSwastodemonstratetheoperationalfeasibilityofcollecting Z longformdataatthesametimeas,butinaseparateprocessfrom,Census2000. @ ThesuccessfulimplementationoftheC2SSduringCensus2000demonstratedthatfull  \ implementationoftheACSisoperationallyfeasible.OperationalfeasibilitymeansthatC2SS B plannedtaskswereexecutedontime,withinbudget,andthatthedatacollectedmetcertainbasic ( CensusBureauqualitystandards.DespitecompetitionfromCensus2000forresourcesandlack  ofexperiencewithanationwideworkload,staffingwassufficient,operationswerecarriedoutas  anticipated,andobservedresponserateswerehigh.Todeterminewhetheranationwide   implementationwouldadverselyaffectoperationalperformance,comparisonsweremade p!  betweenthe1999and2000resultsfrom36counties.BasedontheresultsoftheC2SS, V"! managersareconfidentthatthefullACScanbesuccessfullyimplementednationwidein2003. <#" Specificfindingsinclude: %X $  * #$%&'((xx* #$%&'(" i"   it23  0    TheACSwillimproveplanningandsimplifythe2010decennialcensusdesign. itt݌&$"&(#(# Ќ  " i"   iu23  0    ImplementingtheACS,supportedbythe_MAF_/TIGERmodernization,willpotentially (#( improvedecennialcensuscoverage. iuu݌)$)(#(# Ќ  " i"   iw23  0    TheACShasbeendesignedtocollectdetaileddemographicandhousinginformation. iw3w݌R+&+(#(# Ќ   8,', " i"   i#x23  0    TheACSdevelopmentprogramprovidescurrent,timelyinformationessentialfor  governing. i#xPx݌(#(# Ќ  " i"   iGy23  0    CommunitiescontinuetobenefitfromtheACSdevelopmentprogram. iGyty݌b(#(# Ќ  " i"   i?z23  0    TheACSdevelopmentprogramisimprovingthefederalstatisticalsystem. i?zlz݌. ~(#(# Ќ  " i"   i>{23  0    WorkloadprojectionsfortheC2SSwereachieved. i>{k{݌ J(#(# Ќ  " i"   i%|23  0    AneffectivestrategyallowedtheC2SStocollectmuchofthesurveydatabymail. i%|R|݌  (#(# Ќ  " i"   i.}23  0    TQAandEditFollowupactivitieswerecompleted,althoughstaffinglimitationscaused   somedelays. i.}[}݌x (#(# Ќ  " i "   i[~23  0    TheQAprocessfordataentryensuredaccuratecaptureofmailreturns. i[~~݌D (#(# Ќ  " i "   iY23  0    ComputerAssistedTelephoneInterviewswereconductedonschedule,successfully ` reducingtheworkloadforpersonalvisitfollowup. iY݌F(#(# Ќ  " i "   i23  0    Personalvisitinterviewingwascompletedonschedule,withahighrateofresponse. iӀ݌(#(# Ќ  " i "   i23  0    Totalsurveyresponseratesremainedhighin2000. iށ݌(#(# Ќ  " i "   i23  0    TimelyreleaseofC2SSdataproductsoccurredasexpected. iǂ݌Z(#(# Ќ  NationwideimplementationoftheACSviatheC2SSinCensus2000wasoperationally &v successfulanddemonstratedthatfullimplementationoftheACSisoperationallyfeasible.  \ Basedonthesefindings,theCensusBureaushouldfullyimplementtheACSin2003. B  EthnographicStudies   Therearethreestudiesincludedinthisexperimentalcategory. p!   ComplexHouseholdsandRelationshipsintheDecennialCensusandinEthnographic <#" StudiesofSixRace/EthnicGroups (Schwede2003) "$r# Householdstructurehaschangeddramaticallyinthepastfiftyyears.Inthe1950s,themost %>!% prevalenthouseholdtypewasthe marriedcouplewithchildrenhousehold,alsoknownasthe &$"&  OzzieandHarriethousehold.Asaresultofimportanttrendssincethattime!suchasincreases ' #' inimmigration,changingmigrationstreamsnowcomingpredominantlyfromLatinAmericaand (#( Asia,ratherthanfromEurope,increasesindivorce,remarriages,blendedfamiliesand )$) cohabitingcouples,andchildrenlivingwiththem,andincreasesingrandparentmaintained l*%* householdsandnonrelativehouseholds!householdstructurehasdiversified,andthistypeof R+&+ familyisnolongerthemodaltypeofhousehold.Recognizingthathouseholdstructureis 8,', changingandthatitvariesamongdifferentrace/ethnicgroupsandovertime,theCensusBureau  fundedexploratoryethnographicresearchtolearnmoreaboutnonnuclear,orcomplex,  householdsandtoidentifywaystoimproveenumerationofthem. | ThisstudyidentifiedanddescribedcomplexhouseholdsinselectedethnicgroupsintheU.S. H ThespecificethnicgroupsstudiedincludeKoreanimmigrantsinQueens,NewYork,Latino . ~ immigrantsincentralVirginia,AfricanAmericansinsoutheasternVirginia,ruralnonHispanic  d WhitesinwesternNewYork,NavajoIndiansonanArizonareservation,andInupiaqEskimos,  J knownastheInupiat,inAlaska.Thesestudiescoverfiveofthesixmainraceandethnic  0 categoriesmandatedbytheOfficeofManagementandBudgetforuseinfederaldatacollections.    Thisethnographicresearchprojecthadthreeaims.Thefirstwastoexploretherangeand   functioningofcomplexhouseholdswithindifferentethnicgroups.Thesecondwastoexamine x  howwelltheresponsecategoriesofthedecennialrelationshipquestioncapturetheemerging ^  diversityofhouseholdtypesinthiscountry.Thethirdaimhadthreecomponents:toassesshow D  wellcensusmethods,questions,relationshipcategories,andhouseholdcompositiontypologies *z  describetheemergingdiversityofhouseholdtypes,tosuggestrevisionstotherelationship ` questionandresponsecategoriesforthe2010Censustestcycle,andtocallfornewresearch. F  Complexhouseholdisaresearchcategory,notanofficialCensusBureautypeofhousehold.  Forthepurposesofthisstudy,acomplexhouseholdisdefinedasanonnuclearfamily  household,includingnonrelatives,suchasroommatesandunmarriedpartners;moredistant  relativesnotlistedonthecensusform,suchasnephew/niece,cousin,brother/sisterinlaw; t personssharedacrosshouseholds,suchaschildreninjointcustodyarrangementsandpersons Z tenuouslyattachedtomorethanonehousingunit,andmorethanonefamilysharingahousing @ unit. &v Thereportincludesanintroductionthatgivesbackgroundinformationonfourtopics.Thefirst B isadescriptionofthepurposesforincludingtherelationshipquestiononthecensusform.The ( secondisashorthistoryofhowtherelationshipcategorieshavechangedindecennialcensuses  since1970toreflectongoingchangesinsociety.Thethirdisacomparisonofrelationship  questionsandcategoriesamongmajorCensusBureausurveysandcensusesandthefourth   backgroundtopicincludesanexplanationofhowthehouseholdtypevariableisconstructed p!  fromtherelationshipquestiondataandabriefoverviewofthefivebasichouseholdtypes V"! includedinongoingCensusBureaupublicationsonhouseholdstructure. <#" Theoverallprojectwasdesignedtohaveexperiencedethnographersalreadyimmersedinsix %X $ differentrace/ethniccommunitiesconductexploratoryethnographicstudiesofcomplex %>!% householdsusingthesamemethodsandthesamecorequestionsatthesametimeinthelate &$"& springof2000.Twentyfivecomplexhouseholdinterviewswereconductedineachethnic ' #' communitybetweenMayandJulyof2000,assoonafterCensusDay(April1,2000)aspossible (#( withoutadverselyaffectingongoingCensusBureauNRFUinterviews.AnewAfrican )$) Americanstudywascommissionedin2002asaresultofmethodologicalissueswiththefirst l*%* studydonein2000. R+&+  8,', Fivemajorthemeshavebeenidentifiedinthisresearch.Theyinclude:1)issueswiththe  relationshipquestionandthehouseholdtypevariable;2)cultural,linguistic,andnationality  differenceswithcensusconcepts,methods,andprocedures;3)conceptualdifferencesinthe | definitionandapplicationofthekeycensusconceptof household;4)mobilitypatternsand b respondentsconceptionsofwhoisahouseholdmemberthatmaynotmatchthefundamental H censusresidenceruleconceptof usualresidence;and5)fearandmistrustofthegovernment . ~ andpledgesofconfidentiality.   d  ThemethodofaskingforrelationshipsonlywithrespecttoPersonOnehasthreelimitations  0 identifiedinthisstudy.First,interrelationshipsamongotherpersonsinthehouseholdcanbe    maskedandnotbeidentifiableeitherfromthecensusformitselforinthedataproduced.    Second,theclassificationofhouseholdtypemaychange,sometimesdramatically,dependingon   whoislistedasPersonOne,possiblydistortingthedistributionofhouseholdtypesthatareused x  indevelopingprograms,implementingthepovertydefinition,andallocatingfunding.Third, ^  PersonTwomaynotbethebiologicalparentofacoresidentchild.Thereisawaytoovercome D  theseproblemsresultingfromcollectingrelationshipstoPersonOneonly.Recommendations *z  includedevelopmentandtestingofanindividuallevelquestion,alongthelinesoftheEngland ` censusformquestiontoidentifyallinterrelationshipsinthehousehold. F Thenumberandtypesofrelationshipsthatarespecifiedinstandaloneresponsecategoriesset  limitsonthetypesofcomplexhouseholdsthatcanbeidentified.Therelationshipcategories  usedbytheCensusBureaureflecttherelationshipsinsocietydeemedmostimportantto  specificallydelineateatthetimeofeachcensusaswellasnormsforhouseholdcomposition. t Thesecategoriesexpressrelationshipsbasedonkinship,marriage,andcohabitation,andon Z economic(e.g.,_housemate_/roommate)and/orlegalties(e.g.,adoptedchild,fosterchild).These @ categoriescananddochangeovertime,reflectingchangesinsociety.Indecennialcensuses &v since1970,thenumberofrelationshipcategoriesprintedontheformhasincreased,improving  \ theabilitytoidentifyvariationinlineallyextended(multigenerational)households.Additional B categoriesareneededtoenableusinthefuturetoidentifylaterallyextendedhouseholds(with ( brotherinlaw/sisterinlaw,nephew/niece,uncle/aunt,andcousin,asthesetypesofrelatives  appeartobemorecommonamongnonWhiteethnicsubpopulations,whicharegrowingrapidly.  Resultsfromthisethnographicstudyofcomplexhouseholdsinsixrace/ethnicgroupsidentify   bothlinealandlateralextendedfamilyhouseholds. p!  Theabsenceofdefinitionsorinstructionsforcohabitorsonchoosingproperrelationshipsfor <#" partnersmayleadtoinconsistenciesinmarking husband/wifeand unmarriedpartner "$r# congruentwiththeofficialdefinitions,possiblyreducingthequalityofdataonrelationshipsand %X $ affectingthedistributionofhouseholdtypes.Therearethreefactorsthatmaycontributetothis: %>!% thefirsthastodowiththecategoriesontheformnotfullyoperationalizingtheofficialconcepts &$"& (e.g.,shouldcommonlawpartnersmarkhusband/wifeorunmarriedpartner?);thesecondhasto ' #' dowithdifferentculturalinterpretationsofthemeaningsandconnotationsof husband/wife (#( and unmarriedpartnerinsomesubpopulations;andthethirdhastodowithsocialdesirability )$) andconsequentunwillingnessofsomecohabitingpersonstomark unmarriedpartnerona l*%* questionnaire.Theextentthatrespondentsmarkcategoriesotherthan unmarriedpartner,may R+&+ resultininaccuratecountsofmarriedcouplesrelativetootherfamilies. 8,', ЇRelationshipcategoriesarenotalwaysmutuallyexclusive.Whenmorethanonerelationship  categorycanbemarked,thechoiceofoneortheothermaycausehouseholdtypetovaryand  sometimestobemasked. | Thesecondmajorthemerunningacrossthecomponentrace/ethnicstudiesisthatthereare H cultural,linguistic,andperhapsnationalitydifferenceswithcensusconcepts,methods,and . ~ proceduresthatneedtobeidentified,exploredandtakenintoaccountwhendevelopingforms,  d methods,training,andprocedures.Mostofthebodyofthisethnographicresearchreportwas  J devotedtoethnographicdescriptionsofthesixethnicgroupsincludedinthecomplex  0 householdsstudy.Someoftheimportantcultural,linguistic,andnationalitydifferences    identifiedinthebodyofthisreportthatmayaffecttheaccuracyofcountsandhouseholddata    include:Latinonamingcustoms,Navajomatrilinealkinshipsystemanddifferentkinshipterms,   Inupiaqcustomsofgrandparentsformallyorinformallyadoptingtheirgrandchildren,and x  translationissuesindevelopingforeignlanguageversionsofthecensusform. ^  Thethirdmajorthemecrosscuttingthesestudiesisamismatchbetweenthecensusdefinitionof *z   householdandthedefinitionsofrespondentsindifferentethnicandculturalgroupsthatmay ` leadtomiscountingandmisclassificationofhouseholdtypes.TheCensusBureaudefinition F basicallysaysthatahouseholdconsistsofallofthepeoplewholiveinonehousingunit.The , numberofhouseholdsthereforeequalsthenumberofoccupiedhousingunits.Inthisstudy,  manyNavajoandInupiaqrespondentsdidnotidentifyhouseholdsintermsofsharedphysical  structure,butratheronthebasisofsharingofdomesticfunctions.Emotionalclosenessisalsoa  keycomponentindeterminingwhoispartofoneshousehold.Theethnographersdocumented t casesof householdswithoutwallswherepersonsfrommorethanonehousingunitidentify Z themselvesasonehouseholdaswellastheconverse:peoplesharingoneh_ousing_Ԁunitwho @ considerthemselvestobeseparatehouseholds.Thisambiguityintheboundariesof household &v hasbeendocumentedbyanthropologists,sociologists,economists,andothers.  \ Thefourthmajoroverarchingthemeinallsixrace/ethnicstudiesisthatmobilitypatternscan ( leadtoambiguitybetweenthehouseholdmembershipstatusfromtheperspectiveofthe  respondentsandtheofficialmembershipstatusaccordingtothecensusresidencerules.The  ethnographersidentifiedthefollowingmobilitypatternsforhouseholdsinthisstudy:long   distancecyclicalmobilitytoandfromNavajoandInupiaqhouseholdsfortemporarywagelabor p!  jobs;crossnationalcyclicalmobilitybetweenhouseholdsinLatinAmericaandLatino V"! householdsinVirginiaforwagelaborjobs;seasonalcyclicalmobilityforsubsistenceactivities <#" amongtheInupiatortoescapecoldwintersamongruralWhitescalled snowbirds;mobilityfor "$r# purposesofhighereducation,foundinmostofthesamples;frequentmovementofchildren %X $ amonghouseholdsfortheNavajoandInupiatforschoolingandotherpurposesandforjoint %>!% custodyamongruralWhites;cyclicmovementofelderlypersonsbetweentheirownhousesand &$"& theirrelativeshouses(Navajo)andamonghouseholdsofadultchildren(AfricanAmerican); ' #' sporadicmovementsoftenuouslyattachedpersons(AfricanAmerican);andtemporaryadhoc (#( movesofindeterminatelengthintothehousesofsickand/orelderlyrelativeswhocannolonger )$) manageforthemselves(ruralWhitesandInupiat).Thenatureanddurationofsuchmovesas l*%* wellastheanchorhouseholdrespondentsinterpretationofwhoisahouseholdmembermay R+&+ causeambiguitiesindeterminingwhereapersonshouldbecountedinthecensus.Itis 8,', sometimesnoteasytoapplycensusresidencerulestodeterminewheremobilepersonsshouldbe  counted.  Thefifthandfinalthemewasfearand/ormistrustofgovernmentanditspledgesof b confidentiality.Thisthemeranthroughtherecruitment,completionofmockcensusformsby H respondents,andinterviewingintheNavajo,immigrantKorean,immigrantLatino,andAfrican . ~ Americanethnographicstudies.Thismayrelatetodiscussionsofcorrelationbiasincoverage  d evaluationsresultingfrompersonsbeingmissedinboththecensusandinthecoveragefollowup  J study.  0 Keyrecommendationsaremadeinfiveareas:revisionstoandpretestingoftherelationship    question;newresearchontherelationshipquestionandhouseholdtype;languageandtranslation   issues;outreachandtraining;andnewethnographicresearchrelatedtocoverageandresidence x  rules. ^  RecommendedRevisionstoandPretestingoftheRelationshipQuestion *z   * #$%&'((xx* #$%&'(" i"   i"23  0    Expandthenumberandprecisionofresponsecategoriesintherelationshipquestionto F reflectthegrowingculturaldiversityofthiscountryanditshouseholdcomposition. i"O݌,(#(# Ќ  " i"   i23  0    Addniece/nephew,aunt/uncle,cousin,brotherinlaw/sisterinlaw,andgrandparentas  specificresponsecategoriesforrelativesintherelationshipquestiontobetterreflectthe  rangeofcomplexhouseholds. i݌t(#(# Ќ  " i"   iJ23  0    Add childofunmarriedpartnerasaspecificnonrelativeresponsecategorytoobtain @ moreaccurateinformationonthenumbersandtypesofunmarriedhouseholdswith &v children. iJw݌ \(#(# Ќ  " i"   i23  0    IfconsistencybetweenthecensusshortformandtheACSformisapriorityfor2010, ( modifythenewACSquestiontomatchthedecennialshortform,notviceversa. i݌(#(# Ќ  " i"   iP23  0    Designresearchandconductsemistructuredinterviewsonanexpandedlistof   relationshipterms,developnewwordingforrelationshipterms,particularlyforpersons p!  incustodialcare,andconductcognitivetesting. iP}݌V"!(#(# Ќ  " i"   i23  0    Conductasplitpaneltestusingthreealternativeversionsoftherelationshipquestionand "$r# responsecategoriesinthe2005NationalCensusTest. i4݌%X $(#(# Ќ  ___Recommendations_ԀforNewResearchontheRelationshipQuestionandHouseholdType &$"& " i"   i 23  0    Designnewresearchtodevelopandtestanindividuallevelquestiononamailoutform (#( toidentifyallinterrelationshipsinthehousehold,notjustrelationshiptoPersonOne. i 7݌)$)(#(# Ќ  _" i"   i23  0    Designquantitativeandqualitativeresearchtoassesshowaccuratelytherelationship R+&+ categoriesof husband/wifeand unmarriedpartnerdifferentiatemarriedcouple,_male 8,', householder,andfemalehouseholderhouseholdtypesbycomparingwithactualmarital  status. i݌(#(# Ќ  RecommendationonLanguageandTranslationIssues b " i"   i23  0    Increasethescopeandsizeofthenew LanguageResearchand Translating . ~ DemographicSurveysprojectstoidentifylinguistic,cultural,cognitive,and  d methodologicalissuesindevelopingforeignlanguageversionsofcensusandsurvey  J formsanddevelopandtestimprovedforeignlanguageforms. i%݌ 0(#(# Ќ  RecommendationsonOutreachandTraining    " i "   i^23  0    ExpandoutreacheffortsanddevelopnewoutreachmessagestoimmigrantKoreans, x  immigrantLatinos,andimmigrantsfromothercountriestomaintainandimprove ^  coveragelevelsoftheforeignborninthepostSeptember11thera. i^݌D (#(# Ќ  " i "   i&23  0    DevelopspecialtrainingmodulesforenumeratorsonAmericanIndianreservationsand ` inAlaskaNativeareasthatidentifyculturalfactorsthatmayaffectthewayrespondents F interpretandanswercensusandsurveyquestionsandprovideinstructionsand , proceduresonhowtohelprespondents translatetheiranswersintotheappropriate  CensusBureaucategories. i&S݌(#(# Ќ  2  2"0    PlanandconducttargetedethnographicresearchinotherAmericanIndianandAlaska t Nativetribalareastoidentifyculturalspecificfactorsthatmayaffectthequalityand Z comparabilityofdatawithotherethnicgroupsanddevelopenumeratortraining @ guidelinestoaddressthesefactors.2݌&v(#(# Ќ  RecommendationsforNewEthnographicResearchRelatedtoCoverageandResidenceRules B " i "   i23  0    Planandconductethnographicstudiesofhouseholdcomposition,residencerules,and  coveragebyrace/ethnicgroupsinconjunctionwiththe2006CensusTest. i ݌(#(# Ќ  2  2\"0    Developandconductresearchtoidentifyandassessreasonspersonsindifferentethnic p!  groupsandofdifferentagesmightbemissedinboththecensusandinsubsequent V"! followupcoveragestudiestoreducecorrelationbiasincoverageestimates.2\w݌<#"(#(# Ќ  2  2"0    ConductresearchonLatinonamingcustomsandwhatsurnamestheywriteoncensus %X $ formstoassesstheextenttowhichLatinosvaryinwhichsurnametheyrecordonthelast %>!% namelineandidentifypossibleeffectsofvariationonmatchingandduplicationand &$"& omissionrateswithnonHispanics.2!݌' #'(#(# Ќ  2  2"0    Planandconductnewresearchonpersonswhohavemorethanoneresidenceand/or )$) morethanoneP.O.boxtoidentifyfactorsdeterminingwheretheywishtobecounted l*%* andwhy.2 ݌R+&+(#(# Ќ   8,', Ї GenerationXSpeaksOutonCivicEngagementandtheDecennialCensus:An  EthnographicApproach (Crowley2003)  Thisresearchwaspartofalargerprogramofethnographicresearch,knownasEthnographyfor b theNewMillennium,conductedbytheCensusBureau.Thepurposeofthisprojectwasto H developsocialscienceinsightsthatmayhelpimproveCensusBureausurveyresponseratesand . ~ populationcounts.  d Thepremiseofthisresearchwasthatresponsetosurveysismotivatedbyarespondentssense  0 ofcivicresponsibility.Themaingoalofthisstudywastoinvestigatesharedattitudesamong    GenerationXaboutcivicengagementandcommunityinvolvement,governmentingeneral,and    decennialcensusparticipationinparticular.Participantsofthisstudyweredrawnfrom hardto   reachrespondentpopulations,suchasethnicminorities,lowersocioeconomicclasses, x  immigrants,andalienatedyoungadultswhoareallmembersofthebirthcohortGenerationX. ^  Forthepurposeofthisresearch,GenerationXisdefinedaspersonsaged21to32,thatis, D  respondentsbornduringtheyears19681979.VariousstudiesdefineGenerationXdifferently *z  byage,withsomeanalysescategorizingpersonsbornin1961asthecohortsoldestmembers, ` whileothersuseayoungerupperboundarytodemarcatetheagegroup.Onlyinhindsightwill F theboundariesforthiscohortbecomeclearer.ThewiderGenerationXpopulace,accordingto , paststudies,tendstobeapatheticaboutcommunityandpoliticalinvolvementanddisillusioned  withgovernment.IfGenerationXrespondentsinthisstudysharedsuchattitudesastheirwider  GenerationXcounterpartsdo,thentheCensusBureaufacesanothermajorobstacleinreaching  outtothem.Thisapathyanddisillusionmentwithgovernmentalsocompoundsexisting t enumerationbarriersidentifiedbypastethnographicresearchandmayhaveshortandlongterm Z implicationsforsurveynonresponseissues,undercoveragechallenges,privacyand @ confidentialityconcerns,andeffectiveoutreachcampaigns. &v Researchfindingsandrecommendationsarebasedon150semistructured,individual B ethnographicinterviews,tenfocusgroups,apaperandpencilsurvey,andparticipant ( observationactivitiesindiversesettingssuchasAmericanIndianPowWowceremonies,coffee  bars,communitydemonstrations,classrooms,poolhalls,jobsites,andbowlingalleys.The  scopeofthisresearchwasnationwideandwasconductedinOregon,Illinois,Florida,Texas,   Maryland,Virginia,andWashington,DC.Recruitmentforthisresearchwasnonrandom,and p!  primarilybymeansofsnowballsampling.Recruitmenttargeted25AfricanAmericans,14first V"! generationAfroCaribbeanImmigrants(HaitianandJamaican),20AmericanIndians(onandoff <#" reservations),19SoutheastAsians(Cambodian,Laotian,andVietnamese),59Hispanics "$r# (Mexican,Cuban,PuertoRican,andNicaraguan)and13nonHispanicWhiteAmericans. %X $ Respondentswereprimarilyworkingclassadultswithlevelsofeducationthatrangedfromhigh %>!% schooldropoutstothoseinpursuitofPhDs. &$"&   (#( Thisresearchsuggestedthatthefollowingfactorsmaycontributetodecennialnoncompliance  andundercoverageerrors:  Noncitizenshipstatusorunstableimmigrationstatus b  * #$%&'((xx * #$%&'(" i"   i23  0    Thedistinctionbetweencitizensandnoncitizensisnotnearlyasimportantasthe . ~ distinctionbetweenundocumentedimmigrantsanddocumentedindividualsinanattempt  d toconvincerespondentstocomplywithCensusenumerationefforts. i݌ J(#(# Ќ  " i"   i23  0    Individualswithaninsecureimmigrationstatusweremuchlesslikelytotrustthe    governmentandspecificallylesslikelytorespondtothecensus.Thisresearch    demonstratedthatrespondentswithirregularimmigrationstatusesareunlikelytodirectly   cooperatewiththecensus. i݌x (#(# Ќ  " i"   i23  0    Immigrantcenteredcommunitybasedorganizationsareimportantconduitsthatattract D  distrustfulandgrowingundocumentedpopulationsintheU.S. i݌*z (#(# Ќ  Respondentsnotknowingaboutorunderstandingthedecennialcensus F " i"   io23  0    GenerationXrespondentsinthisstudyvaluethedecennialcensusasimportantand  worthwhile.Whilemostrespondentswerefamiliarwiththecensusbyname,most  respondentswereunclearabouttherolesandfunctionsofthedecennialcensus.  However,mostnewimmigrantsandAfricanAmericanrespondentsinthestudywere t significantlylessinformedaboutthecensus. io݌Z(#(# Ќ  " i"   i23  0    RespondentseitherexpressedacompletelackoffaithinthenotionofCensusBureau &v privacyandconfidentialitystatementsorwereambivalentaboutsuchCensusBureau  \ promises.Inspiteofperceivedrisks,mostrespondentswerestillwillingtodivulge B personalinformation. i݌((#(# Ќ  Increasedlevelsofdistrustamongrespondentstowardsthegovernment  " i"   iN23  0    Skepticismandmistrusttowardsthegovernmentwaspervasiveamongthisgroupof p!  respondents.Respondentspastnegativeexperiencesandinteractionswithfederal V"! bureaucraciesinfluencedtheiroverallnegativeattitudestowardsthegovernment. iN{݌<#"(#(# Ќ  " i"   i%23  0    Althoughrespondentsinthisstudypossessedunfavorableattitudestowardsthe %X $ government,derogatoryviewswerenotextendedtowardstheCensusBureau. i%R݌%>!%(#(# Ќ  " i"   i23  0    Respondentswerestillwillingtocomplywithdecennialenumerationeffortsbecause ' #' theybelievedthesocialimportanceandbenefitsofthecensusoutweighdistrustful (#( attitudesheldtowardsthegovernment. i݌)$)(#(# Ќ  " i"   i423  0    Respondentdistrustofpoliceandlawenforcementagenciesisalinkthatservesasa l*%* deterrenttoCensusparticipation. i4a݌R+&+(#(# Ќ   8,', Miscellaneousfindings  " i "   i23  0    MinorityGenerationXrespondentsinthisstudygenerallydidnotconfirmmanyofthe | negativestereotypesthatsurroundallGenerationXcohortmembers. i݌b(#(# Ќ  " i "   i23  0    Family(includingfamilyvaluesandfamilyrelationships)isseenasthemostimportant . ~ socialinstitutionamongthisagecohort;familyseemstorepresentthemoststablesense  d ofcommunityforGenerationXrespondents. iK݌ J(#(# Ќ  " i "   i23  0    Religiousactivitieswereextremelyimportantamongallrespondentgroupsinthisstudy    withtheexceptionofnonHispanicWhiteGenerationXrespondents(asmallpercentage    oftherespondentsinthisstudy).Fornoncitizenimmigrantrespondentsinthestudy,the   churchanditsparishionersrepresenttheirlocalcommunity. i݌x (#(# Ќ  Therecommendationsthatfollowarebasedonsound,empirical,ethnographicresearchprovided D  totheCensusBureautohelpmakeinformedplanningandpolicydecisionswithregardstothe *z  2010decennialcensus.Keyrecommendationsare: ` " i "   iE 23  0    Downplaythenationalnatureofthecensus;emphasizemakingadifferencethrough , Censusparticipationinoneslocalcommunitywithoutoverpromisingbenefits. iE r ݌(#(# Ќ  " i "   i 23  0    Stressbenefitsofthedecennialcensustofamilies;promotethecensusasafamily  activitythatwillalsohelponesethniccommunity. i  ݌t(#(# Ќ  " i"   i 23  0    Continuetopartnerwithchurchandfaithbasedorganizationstoreachspecialpopulation @ groups,especiallyimmigrants.Recruitandhirechurchyouthgroupsforoutreachand &v enumerationwork. i 6 ݌ \(#(# Ќ  " i"   i 23  0    ConductfurtherresearchtodetermineiftheCensusBureaushouldonlyemphasizethe ( distinctionbetweencitizensandnoncitizensinitsadvertisementsandoutreacheffortsto  variousrespondentgroupsoriftheCensusBureaushouldalsoemphasizethatpersonal  informationisconfidentialfordocumentedandundocumentedresidents. i  ݌ (#(# Ќ  " i"   i23  0    Continuetofocusonallundocumentedimmigrantpopulationgroupsineducationaland V"! outreachcampaigns. i)݌<#"(#(# Ќ  " i"   i/23  0    Publicizethatthedecennialcensusisaneasywayinwhichagenerationcangivebackto %X $ thecommunitywhileempoweringthecommunity. i/\݌%>!%(#(# Ќ  " i"   i23  0    CollaboratewithImmigrationandNaturalizationServicestoincorporateoneortwo ' #' decennialcensusquestionsontheNaturalizationCivicsExamsponsoredbythe (#( DepartmentofJustice,ImmigrationandNaturalizationServices. i݌)$)(#(# Ќ   l*%*  PrivacySchemasandDataCollection:AnEthnographicAccount (Gerber2003)  Thisethnographicresearchexaminedabroadrangeofrespondentconcernsaboutprivacyand | confidentiality.Itexaminedthefactorsthatrespondentstakeintoconsiderationwhentheyare b askedtorevealinformationaboutthemselvesacrossavarietyofvenues.Theaimofthis H researchwasessentiallydescriptive. . ~ Atotalof120interviewswerecarriedout.ThirtynineinterviewswerecarriedoutinPhaseI  J withrespondentswhohadparticipatedinatleastoneCPSinterview.Anadditional81  0 interviewswerecarriedoutinPhaseII,withrespondentsrecruitedbylocalorganizations(many    ofwhichhadbeenpartnershipgroupsinCensus2000)andbyothercontacts.OverbothPhases,    37nonHispanicWhite,21AfricanAmerican,17AmericanIndian,14Asian,3PacificIslander,   23Hispanic,and5respondentswhoselfidentifiedasmorethanoneracewereinterviewed. x  Semistructuredresearchprotocolsweredesignedtobeadministeredbyateamof D  ethnographers.Theinterviewusedflexibleprobes.ThetopicsincludeddebriefingaboutCensus *z  2000andCPSparticipation,experienceswithotherdatacollections,privacyattitudes,anda ` seriesofvignettes.Thesevignettesservedtoexpandthesetofcircumstancesunderdiscussion F toincludethingsofparticularinteresttotheresearch. , Adescriptivemodelforunderstandinghowrespondentsdecidewhethertodivulgeinformation  wascreated.Thismodelincludedthreemainparts:anassessmentofthesponsorofthe  questions,anassessmentofwhetherthequestionsarerelevanttosomelegitimatepurposeofthe t sponsor,andanassessmentofrisksandbenefitsofdivulginginformation. Z " i"   i23  0    Inassessingsponsorship,respondentswanttoapproveofthesponsoringagency. i݌&v(#(# Ќ  " i"   i23  0    Respondentsarealsoawarethatsponsorshipmaybemisrepresented;thus,the B authenticityofthedatacollectionisanissueforrespondents.Thismakestelephone ( modeinterviewshighlyunpopular,becauseitisimpossibletobesureofacaller'strue  identity. i݌(#(# Ќ  " i"   i23  0    Respondentsformexpectationsofwhatquestionsarelegitimateforthesponsortoask, p!  basedontheirunderstandingofthenat_ure_Ԁandpurposeofthesurveyandthesponsoring V"! organization.Questionsthatgobeyondthisframeworkofexpectationmaybeperceived <#" asintrusive. i݌"$r#(#(# Ќ  " i"   i!23  0    Respondentsevaluatetherisksandbenefitsofprovidinginformation.Allrespondents %>!% arefamiliarwithexchanginginformationtoreceiveparticularbenefits,forexample, &$"& providinginformationtolendersortosocialserviceagenciestoreceiveservices. ' #' Respondentsarealsomotivatedbyaltruisticbenefits,suchasprovidinginformationto (#( thedecennialcensustoenableservicesforalocalarea.Theyalsomayseeparticipation )$) insurveysandcensusesasawayofbringingagrouporapointofviewgreaterattention. l*%* Thisiscalled havingonesvoiceheard.ItwasapowerfulmotivationforLatinosand R+&+ AmericanIndianstoparticipateinthecensus. i!"݌8,',(#(# Ќ  " i"   i%23  0    Respondentsalsoworryabouttherisksofdivulginginformationaboutthemselvesand  theirfamilies. i%%݌(#(# Ќ  " i"   i&23  0    Respondentsarealsoconcernedaboutgovernmentagencies,suchaspolicedepartments, b theInternalRevenueService,theImmigrationandNaturalizationService,and,for H AmericanIndianrespondents,theBureauofIndianAffairs.Theseagenciesarenotseen . ~ asbenign. i&'݌ d(#(# Ќ  " i"   i(23  0    Becauserespondentsbelievethatinformationisfreelysharedbetweenagencies,despite  0 anyassurancesofconfidentiality,iftheyhavesomethingtohide,theyarereluctantto    provideittoanygovernmentagency. i()݌  (#(# Ќ  Thismodelofhowrespondentsdecidetorevealinformationincensusesandsurveysiswidely x  sharedinallgroups.Howeversomeprobabledifferencesdidemerge.Onedifferencewas ^  betweenmoreandlesstechnologicallysophisticatedrespondents.Technologicallysophisticated D  respondentsweremorecomfortablewithprovidinginformationontheInternet,andfeltmore *z  abletodealwithanypotentialproblemsthatmightoccur.Simultaneously,however,such ` respondentsoftendidnotbelievethatitwaspossibleforanyinstitutiontocompletelyassure F privacyorconfidentialitytopersonsprovidinginformation.Differencesinprivacysensitivities , alsoemergedforgroupsthathadnegativeexperienceswiththeFederalGovernmentorwith  particularagenciesofgovernment,(e.g.,legalimmigrantswhoexperienceddifficultyincrossing  theborder).  Recommendationsbasedonthisresearchinclude: Z " i"   i.23  0    Itisnotpossibletocreatealistofitemsthatwillalwaysorneverbeconsideredprivate &v becauseprivacyjudgementsaresituational. i.,/݌ \(#(# Ќ  " i"   iR023  0    Becarefulhowdisparatetopicsarecombinedinonesurveysettingbecausethesenseof ( intrusivenessofquestionsissituational. iR00݌(#(# Ќ  " i"   i123  0    Includetheideaofhavingonesvoiceheardinmotivationalmaterialforminority   groups. i11݌p! (#(# Ќ  " i"   i223  0    DescribetheCensusBureausinternalcontrolsonthehandlingofdatainexplanationsof <#" confidentiality. i22݌"$r#(#(# Ќ  " i"   i323  0    Providegood,understandableexplanationsofwhydataareneededandhowtheywillbe %>!% usedbecauserespondentscomfortwithquestionsrestsontheirassessmentofthe &$"& sponsorslegitimaterighttoknowtheinformationrequested. i3,4݌' #'(#(# Ќ