WPC  8Z,v^:܆񔦵ڻhѷ *B0}AwC|3N۸b-#1ȃKsp56e8H4U8k;8Ď+u%+lnKc3^!^[8F!T/?s<[7 HXOh[6n"l$,%%T3^࿉1}v(lv-KemGJ`;,Z*g,uєFf_bDg3*eTvr.,=>ǯ}1eoBFZ}$@vyV# UN %! 0:' ^ a wm 4q   m U B 0~ 0DmE 0uc 1u, 72 0w 0J 0 0t 0( 0 0 0D 1 72U>NU+NU>FMU*MMMMMMMMM9b@J(a/9Mo?``$Jn}u% 0N@BBBBBBBBB BD D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D  0Va a a  A  D+V  0K  \\CENSUS\PRED-TR2-2-HP9000-1A.PRED.DECENNIAL.HQ.CENSUSdttddtdt0(9 Z6Times New Roman RegularX($USUS.,'/ s.:i+003|x"U\  `*Times New RomanTT(.(3($ !USUS.,      0  (#$  0    (;3$2#  0  .3  0  3#37=CIQYag1.a.i.(1)(a)(i)1)a)i)(O;$0  2#  a  .3  0` (#(#(b$0  0` (#(#2#   .3  0 ` (#` (#(xir$0  0` (#(#0 ` (#` (#2#(  0  )3  0 (# (#($0  0` (#(#0 ` (#` (#0 (# (#2#(  a  )3  0h(#(#(F$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#2#(   )3  0h(#h(#($0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#2#  0  )3  0(#(#({$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#0(#(#2#  a  )3  0p(#(#(F$0  0` (#(#0 ` (#` (#0 (# (#0h(#(#0h(#h(#0(#(#0p(#(#2#     )3  0p(#p(# xAGaeimquy}Bullet ListBullets List\  `&Times New Roman(hH  Z6Times New Roman Regular(hH  Z&Times New Roman%2A`Arial /#`Xd#  - -3H:\MURAWSKI\LATESTMO\1CPOSTAG.TIF 78dJJdskJeffdkline WPC   !!!"""###$$$%%%&&&'''((()))***+++,,,---...///000111222333444555666777888999:::;;;<<<===>>>???@@@AAABBBCCCDDDEEEFFFGGGHHHIIIJJJKKKLLLMMMNNNOOOPPPQQQRRRSSSTTTUUUVVVWWWXXXYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffggghhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuvvvwwwxxxyyyzzz{{{|||}}}~~~o}sϏO/??oL߯O? ?O??O???̅{3OD_/_3߬.7_/o.ߓ3o/_2;)ߜ?A?>2_?_O/+.? ?5_/o+_?1oo//_7) O:+?AO=?0/=?$o-+?._ O??4+0?/"4?)$_8+o@o=??O/;?'?.+?. ?3_,/?/$3?O*&_6+@=?OO. /? ?: O_/?.?*?. 2_ _o/ ,.o Oo? o0&_2O*/&6+@<O- O_?8_  _ .O*?.o1_? ,??- O 0&_1*O'5*? ?<￟? ߿_. ?7/ ??0Ͽ? ߂0o __2/ _-߿/ ?Ͽ. 2 ￿O 1￟? ߿_.Ͽ? OO 6 ς?_ <￟? ߿_ ??0??6/ ?2 o3 /3/O0 0o4? _ 3 ??2  7? ?> o? ?? ?O/__5/ /2 3 3?0 /0o/4? ? ?3 ?O2 / _7? ?> ? ?_ _0O5O _?2 3 20 ?/?4 _ 2 _2 / 6 > > O /4 _?2 3 ?3?/?0 ??/_o4 ? _2 3  _6 = O>  /?/3 ?2 ￏO?3? 2_1 ??/o4  2 3  5 ??=_>  .?3/ 2 Q? ?21 ?./4 _ 2 3 ? 5 Z>  ?.2 ?/2 Q????2?߿1 o.o￯4  2 ?3  _5 Z> ??.oM o$3 Qo??2?M .?Q  O2?4  /5 Z/?? =???.?N [ Q??1?N .?Q /?2??4 ? 5 Y O=????.?M \ QO??1?N .Q ??2??4 O 5 Y_ <????.?L \ Q/ ??1/N . Q ??2??4  4 X__ <????.?L/ o] Q? ??1 M . P ?O2??4  4 X <????.?/K ] Q?? o??1 M . oP ?2??4  4 X?// ;????.O J O^ Q? ??2 M .O 2??4 o 5 W O;????. ?J/ ^ Q? ??2 L .? oO  2??4 ? 5 Wo :????. H ^ Q?/ ??2_ K . N O3??4 /?5 W__ :????/ G ?_ Q? ??2 J . M _3??4  o5 V :????/ Fo _ Q? /??3 I /? L /4??4  5 V? 9????/ _D/ _ Q? ??3 H / K o4??4 _6 U _9????0 C ` Q? ??4/ OF 0O ?I o5??4 _6 U 8????1/ o@ ?` Q?o ???4 E 0 H /6??4  6 UO _ 8????1 ? _` Q? ??5?C 1F //6??4 _7 T  8????2= ` Q? ??6__A 2?D /7??4 O7 TO  7????3?; ` Q?O O??7_? 3oB __8??4 _ 8 So  _7????49O ` 8? ??8o> 4A ߿/:??4 _ _9 :  6????5/8? a!8?  ??9?< 5/?_;??4 / ?:!:? O 6????6_?6? a!8? ? o ??:/: 6?_=_=??4 ? O:!9  /6????85? a!8?  ??;o9 8<????4 ? o<!9_ 5????9_3? a!8?   ??=/O7 9??:o<??4=!9o _5????;/2? a!8? /  ???6 ;9:??4o>!9 4????=?0? a ߃9?  ??@_5 <_7 O/8??4o@ σ9/O 4????>/? a Q?  / ??B?_3 >O/6 o O6??4B P /4????@O.? a Q?  ??D/2 @?/5 _ 5??4?B P_ 3????B/- a Q?  ??F_2 B/4 ? 4??4  A O_o _3????D /, ` Q?o ???G_1 C_4  3??4 o @ O 2????Eo o, ` Q? ??I/ 0 E? 3 / 2??4  ?@ O/? 2????G + ` Q? ??K ?0 G 3 / O2??4  ? N ?2????H/ _+ O` Q?O O??L / H 2  1??4 ? > No 1????I_ +` Q? ??M O/ I/ 2  1??4  ?> N(o1????J/ +O _ Q? ??N . J 1 O 0??4 o = M)1????K O+ _ Q?? o??O . K 1  _0??4  < M?*0????L + O_ Q? ??P? o. L_ ?1 /0??4  O< L+?0????M + ^ Q? ??P ?. L 1  /??4 ? ; Lo,/??OM +_ ^ Q?/ ??Q./MO 1  /O4  : L?? o/_oM + /^ Q? ??Qo .?M 1  /4 _ O: K /o?N , ] Q? /??Q .OOM 1  /?4  9 KO? .??N, ?] Q? ??Q .o?O 1  /?4  8 Jo ?./ ON?, \ Q? ??Q.?O 1  / O4 / O8 J - OO?- \ Q?o ???Q?.OO1  05  7 J/o ./OOO. [ Q??Q_/N?1 / 0OO5 _ 6 I .OO_._?/ Z Po/?1o/Oo.ooo1  0_5  O6 I ? -OO.0/ X 1?O?1?O///??o1  /1O4  5 )_! ?- O/O/_0 /W _?1??1?/0/?2  2/O5 / 4 O?(?" -/O_/_?2 /#1 1??11_?/?2 O 2_5  /4 '# ?-_O?0?_3_ / ?1 1? ?12 o02  _4O?6 _ 3 '?o# -O? 0?o?4/ 1 1 ?1O3_ 0o3o  4 6  2o ?&$O O- ? o1?_5/ o1 o1 ?0_?413? O 6? o7  1? ?&?/% -? O _1?7/ 1 /1/ ?0?_5 oO24?  _ ?7 _7 / O1? %& _- _  ?2 O _9/ ߏ/-Ͽo ? O-_? ?0 ? 7 ?/2 ? ? 1߿ Ͽ_ /8 ?4￟/ o ?(ϯ_ ?/ "/_߿/ o+ ?  ? 3 o_ _;_/oￏ_ -?,?,o ?0 /? 9_ ?ߏ2 O 2(/: ? 5?o_(-"O o+ ?  OO/ 4?? >?/-/,?,_ ?0/_o? ; /_O 3Oo? 3'o= OO/ 6?O(-"? ?, O_/  /5!@O$/-?,? ?0!O=?4!/3%?/7/?'-" /-/ o6oC? 0-?,__?0@o65#?Co7 '-O"/o _9ϏO_G__2-,??1o?C_o9?6/_F_9'-?!o0__/?ϟo?/Lo/?5﮿,ϐ?6O/_HOO?_/O9ߝN_/;'߭!ߒ3_/ϟO?/?_G߿??/?ORo????_Ƈ;o???_OϏ????oHϯo???OϟO?/?_lϟO?/?_AJPO?[?_o/Oo?P/[_O Po[??/   $.O]  TO)F 0ooo/?_No%?c o  O&/-] _ TO+G O/?/L??$c  &/.] OU*OG .?Jo$c /'o._oWO,I0Ooϯ/H$c O(.O/`/_YO-I?0/ _G$Od ?(.a_o o?Z?-J0??O F%d ?(o /Z-??JO0o/ /)??O [oo.oJ/0O_?)? /\?O/o??-J/0_O?o(O] /o-I_0o__O ?/o _?߃?/o//__?/o? ߏ?/ _?o?? /o/?oO?? ?ߏ?O?O o_ o?? ߂o?__O?_ _o//?/O//o?ߏ?__/Oo? O?o _?_/O _?o___/O/_o o /??_ ?  ?_/  ?? ??__    OO_O__?/ _ O  o? ? ?_?/ ? ??? ??? _?O_ O /_o_? O? _ o _ // o  /  O?_/ oO   ?/ ??/  / ? ???? oo/ oo/ o_? __/?/ _ߟ/_? ?_?oO   O_/ o/ o o/o/  o _ OOo?? //_ _?߯_/o _/o __? _ ? _o /  ?oO  o / o_ ?/? ? /o ? ? ?o_ //? __ _//? _oO /o_? /? /oo/??  ? ?O/ _ /  oO ? O/  O o/ _  _/ o  _/? o//_? _ o??oo_? _ / Oo/_? /?O? ? /_O //? ?/ O o /?     _ ? _ ?OO_o   ?// ?_o_ /o_   ?/߿/?/_  / O??/? O  / /o _ o_O/ O//  ?? O_   !_/ / _ Oo/o   _O ?/ ? _O _ o/ ??// O _/ O?/_   o    /_?? O߿? /_ _ o/ ?_? O o" O ? ?/_? /? O/ o O/  /?   //O/  // O? __ o /  ?ooO _  ??#/O _ ? /o  _ /? ?_ O_   ?O_Oo_ ?  / _ ? O ?_O/?  /?O /O  #  ??? /? ?   _o   oo O OoO_? /O? O_O/ _/ o _o ?_ _ # / ?o O?o?O _  ?/OOo_ ?/??  _ _?o?/??__  ??_? ?? ? Oo _O? ?/  // / #o O  ?o   O?O_O//  _  //__ ? OO_o/_/_  ?? /_?_/o _ oo  ?/ O$   ?O? _o ? ?/  /?  /  _// _/ o/ oo  _ ooo/  ?/ _  _  ?$   o O  ?_ OO   _? OO _?ߏ _/ O_ o _ _/o ?? _ _oO  _? /? ? /O o$? O/ ?/ O_O  o_??   o  o OO?o _?_/o  ?o?? / /? O  /o /o   o  O ?$? ? O__??/O  ?/O? _"_  ?O   _  /O   ??_ _ ?// o /o o ???o? ?    _ # o _?oo o O/O%/ ?  _  ?? _ ?_O O_??oO/  /  ?? o?O o /  ? o$  /o?O?_  /+/   /?  _ _ /? _O/o/?/  O  o ?? ? _ _ _/O?$o  _?oo_? ?O ?+_ O    O  _/? oO   _ o o _ //  /__O?o$/  / ?Oo_?//  ?+ O _   _ O  _o/O?? ?/?  _ //__o   _  ?_$? ? O O///_ ? o_  /O    _  /OOoO??//o??_ O_ / /? o/ / /  ??  o ?#_  o /__O ?_ O?/  o/   _ O  ? O  ?/_?/ ? oo_??o  ?  /?O O/  _?? o /  // ?O$ / o?/ ? ?/ __o O O/  o  /? o o O   O  o / _? ?/   O /O _ ??  ? _ ? ?O/O/$? _O/ o O O ?_/  _?? /  _ /  o O?? ?/ OO/?o? oo? o ?o _ O/    _oo/_$ / ? ߿  ?_/? OOOߏ/ O ? _ oO  ? oOO_?ߟ?o ?/?O/oO// _O / ?_ _ ??????O?   /!o O_ߟ/O  oO/oO? _o/??? o/Ϗo / _ oO? /o?o????/o??__oo ? ?OO/      ? ?  _!O / ? ?O ?_  o _  ?  /  o /?_ _??/_   ?  /?? / _   _ o_?/ Oo  ? oo O? _?  o ? /?/ ? O  ? ? //? o  O  ?  o  _   //_ ?O/?߄  O?/!ߊOO/oτ?o ?OO?_  _? o?/_ ߄  OO?o?oO _?O  //o ?? 4߿%_V _@aD\߿m߿`Ͽ*Ͽ)Ͽ@\bi\Ooj_\_Poj?\?Q /jo[oQ_? j[Qoko[oQ??l/[/Sn?Z?S_ /noZX߯oq???o?????/??_O??/?__????????_??o????o?/???/???????_/???o??__???????/OO?o??_????/o??o???/??O_????_????//??_???????__?/??OO??????OO??_o__??_o_????O???o߿/??__????/?????/_??/??O????_????o??/?__???O/????????o????o_????_????/O??o??o_????/??/????o??????߿???__O/??oϿ/O??????/???_O/oO??/????O????/?/_???_????/???_/???_o_????/?o???__??o????oo???/?/???/?????????/O???????/_O?OO?????o????_???O/???o????/O???O??o????O??o??_???????O????o_?/??O????????????o??O_?/??/OO??o_o/???o?_Oo??/??OO?O???????o??o?/O?????O/OO??o????O/o/OO??_?/OO_OO__?o/?oOoooo_O?OO/?O??O/??o/O_?O/O_/_????/?/O/O??/O__?/??_??O_/?_O??__/???o_O?_?oO??o?/?O_ooo?OO?o?_/oo?_??O?o???/?O_?///??_oO?_?_/O?__?O_/ߏ/Ͽo?????????????????O_????/??߿Ͽ_/?￟/o?ϯ_????????????????//_߿/o??o___/oￏ_??o?/?_?ߏO/??o_Oo?OO/??????//?_?/_o?/_OOo?oOO/?O??O_//O???O?///?//oo??__?o?oOo_ϏO___??o?_o?/__?o__/ϟo?/o/?￿Ͽ?O/_OO_/O߿_/￿߿￿߿_/ϟO?/?_߿??/?Oo????_o???_Ϗ????oϯo???OϟO?/?_ϟO?/?_O??????????_o/Oo?????/_Oo??/OOooo/?_o?oO/_OO?/??/OO?o/ooOOoϯ/OO//_O?/_O?_oo????O?o/??Oo//??Oooo/O_??/?O/o??/_O?oO/o_o__O????/o_????????/o//__?/o?ߏ?/_????o?????/o/?oO?????ߏ?O???Oo_o??o????__O???__o//?/O//o?ߏ?__/Oo????O?o_????_/O_???o___/O/_oo/??_??_/????__OO_O__?/_Oo???_?/???????_?O_O/_o_?O?_o_//o/O?_/oO?/??//?????oo/oo/o_?__/?/_ߟ/_??_?oOO_/o/oo/o/o_OOo??//__?߯_/o_/o__?_?_o/?oOo/o_?/??/o???o_//?___//?_oO/o_?/?/oo/????O/_/oO?O/Oo/__/o_/?o//_?_o??oo_?_/Oo/_?/?O??/_O//??/Oo/?_?_?OO_o?//?_o_/o_?/߿/?/_/O??/?O//o_o_O/O//??O__//_Oo/o_O?/?_O_o/??//O_/O?/_o/_??O߿?/__o/?_?OoO??/_?/?O/oO//?//O///O?__o/?ooO_??/O_?/o_/??_O_?O_Oo_?/_?O?_O/?/?O/O???/??_oooOOoO_?/O?O_O/_/o_o?__/?oO?o?O_?/OOo_?/??__?o?/??__??_????Oo_O??////oO?oO?O_O//_//__?OO_o/_/_??/_?_/o_oo?/O?O?_o??//?/_//_/o/oo_ooo/?/__?oO?_OO_?OO_?ߏ_/O_o__/o??__oO_?/??/Oo?O/?/O_Oo_??ooOO?o_?_/o?o??//?O/o/ooO???O__??/O?/O?__?O_/O??__?//o/oo???o??_o_?oooO/O/?_??_?_OO_??oO//??o?Oo/?o/o?O?_///?__/?_O/o/?/Oo???___/O?o_?oo_??O?_OO_/?oO_oo_///__O?o//?Oo_?//?O__O_o/O???/?_//__o_?_??OO///_?o_/O_/OOoO??//o??_O_//?o///??o?_o/__O?_O?/o/_O?O?/_?/?oo_??o?/?OO/_??o///?O/o?/??/__oOO/o/?ooOOo/_??/O/O_???_??O/O/?_O/oOO?_/_??/_/oO???/OO/?o?oo?o?o_O/_oo/_/?߿?_/?OOOߏ/O?_oO?oOO_?ߟ?o?/?O/oO//_O/?__????????O?/oO_ߟ/OoO/oO?_o/???o/Ϗo/_oO???/o???o????/o????__oo??OO/??_O/??O?_o_?/o/?__??/_?/??/__o_?/Oo?ooO?_?o?/?/?O??//?oO?o_//_?O/?O?/OO/o?????o?OO????__?o?/_OO?????o?oO_?O//o??߿__߿߿ϿϿϿo__o??/oo_?ooo?????//??_/o߯o.@DH:\MURAWSKI\LATESTMO\1CPOSTAG.TIFH:\Murawski\Latest Models\1cpostag.tifT4b̤"ͤͤnJ' ']&I{s u&      ($USUS.,    1    _OMRtechnologyusesanopticalscannerandcomputersoftwaretoscanapage,  recognizethepresenceofmarksinpredesignatedareas,andassignavaluetothemarkdepending  onitsspecificlocationandintensityonapage;OCRtechnologyusesanopticalscannerand  computersoftwareto readhumanhandwritingandconvertitintoelectronicform.   ($USUS.,    2    _ActualnumberswerereportedafterthecompletionofCensus2000.Thefinal_PMP_  issuedatewasMarch30,2001.   ($USUS.,    3    __RITRC_ԀdefinesacceptancerateasthefractionoffieldsinwhichtheOCRhashigh  confidence,usuallyexpressedasapercent.Acceptedfieldsaretheones_RITRC_ԀscoredforOCR  accuracy;theyarenotsentto_keyers_ԀexceptforQApurposes.   ($USUS.,    4    _Fullfieldkeyinginvolvesreenteringtheentirefieldasopposedtocharacterkeying  whereonlythelowconfidencecharactersinafieldareentered.   ($USUS.,    5    _AccordingtotheDataCapture_PMP_,theDCS2000contractoroperatedtheBowieTest  Sitewhichhousedprototypesofthesoftwareandhardwareenvironmentsusedinthe_DCCs_.  System,integration,andsecuritytestingandtroubleshootingforDCS2000softwarewas  performedattheBowieTestSitebythecontractor.(O$http://www.census.gov/pred/www/  ($USUS.,    6    _AlsoreferredtoasParetosLaw,thisistheprinciplethat20percentofsomething  accountsfor80percentoftheresults. d(.3$ !USUS.,   I&mage <=8C HKKKK'dxd( $ Figure  1  Level 1Level 2Level 3Level 4Level 5 !USUS.,  _OY,UXXXXTRX3'LetterX3' Letter3'Letteretter3'LetterT 8XXdd8-  #XX~# Q   Census2000Testing,Experimentation,  andEvaluationProgram0 h 0h(#h(#0(#(#0p(#(#0p(#p(#0 (#(## Q#&%July22,2003#%&# Q& (# (# __________________________________________________________ s    `     h        `     h      p # Q#XX TopicReportSeries,No.3  # s #XXh#   `    = CENSUS2000 ;  DATACAPTURE# =2#   XXQualityassuranceprocedureswereappliedthroughoutthecreationofthisreport. _ Thistopicreportintegratesfindingsandprovidescontextandbackgroundforinterpretationof 7 resultsfromCensus2000evaluations,tests,andotherresearchundertakenbytheU.S.Census # Bureau.Itispartofabroadprogram,theCensus2000Testing,Experimentation,andEvaluation  program,designedtoassessCensus2000andtoinform2010Censusplanning.#XXȷ#     `     h      `     h    @  @`  @  @  @h  QPreparedbytheTitanCorporation $g # Q # AU) xdEP%xA Q%     @  @`  @  @  @h  QKevinA.Shaw,ProjectManager `%  0 @ 0@` (#(#0@ ` (#` (#0@ (# (#0@h(#(#Planning,Research,andEvaluationDivision &! # Q #  (#(#(#(#n.:*&`z00($  `(*E,$ `$ `e, n#T-("(#(# #ЇXXIntentionallyBlank  %! ,U ,8XXdXXd8 ,     XX 8XXdXXd86Contents     XX 1.0  INTRODUCTION................................................................................................................1&(#(# 2.0  BACKGROUND.................................................................................................................1(#(# 3.0  SCOPEANDLIMITATIONS.............................................................................................4 (#(#   4.0  DATACAPTURETOPICSADDRESSEDINTHISREPORT.........................................6^ (#(#   4.10 ` Performanceofthedatacapturesystem...................................................................6J  ` (#` (# 0  4.20` (#(#Thesystemsabilitytocapturequestionnairedata..................................................86 ` (#` (# 0  4.30` (#(#Theimpactofdatacapturerequirementsonthequestionnairedesign"r ` (#` (# Ѐ0  0` (#(#andotherfactors.......................................................................................................9^ ` (#` (# 0  4.40` (#(#Theappropriatenessofrequirementsidentifiedforthedatacapture J  system.....................................................................................................................106 ` (#` (# 5.0  FINDINGS.........................................................................................................................11(#(#   5.10 ` Assessingtheperformanceofthedatacapturesystem..........................................11` (#` (#   5.20 ` Factorsaffectingthesystemsabilitytocapturequestionnairedata.....................16` (#` (#   5.30 ` Issuesrelatingtotheimpactofdatacapturerequirementsonthequestionnaire  designandotherfactors.........................................................................................18n` (#` (#   5.40 ` Examiningtheappropriatenessofrequirementsidentifiedforthedatacapture Z system.....................................................................................................................19F` (#` (# 6.0  RESULTSOFANALYSIS...............................................................................................21n(#(#   6.10 ` Systemperformance...............................................................................................21 Z` (#` (#   6.20 ` Capturingrespondentdata.....................................................................................22F` (#` (#   6.30 ` Theimpactofdatacapturerequirements...............................................................222` (#` (#   6.40 ` Fluidrequirementsposedsubstantialrisks............................................................23` (#` (#   6.50 ` OthersalientobservationsaboutCensus2000datacapturesystem......................230@#` (#` (# ` (#` (# 7.0  RECOMMENDATIONS...................................................................................................26!(#(#   7.10 ` Unifythedesignstrategyneededforthedatacapturesystem...............................26~" ` (#` (#   7.20 ` Definerequirementearly........................................................................................27j#!` (#` (#   7.30 ` DevelopQAstandardsearly..................................................................................27V$"` (#` (#   7.40 ` Focusonredesigningkeydatacaptureproblemareas...........................................27B% #` (#` (#   7.50 ` Limitthenumberofforms.....................................................................................28.&~!$` (#` (#   7.60 ` Assessfutureroleofautomateddatacapturetechnology......................................28'j"%` (#` (#   7.70 ` Implementaunifiedhelpdesk...............................................................................29(V#&` (#` (#   7.80 ` Trainontheactualworkingsystem.......................................................................29(B$'` (#` (#   7.90 ` Expandthetimefortesting....................................................................................29).%(` (#` (#   7.100 ` BetterdefinitionDMISforfuturecontracts...........................................................29*&)` (#` (#  XXXX   7.110 ` Providemoreinformationonthescopeofdocumentationrequirements..............30+'*` (#` (#   # XX##XX 7.120 ` MinimizetheuseofKFP.......................................................................................30` (#` (# # XXȮ$#XX   7.130 ` Producerealtimecostdata....................................................................................30# XXȏ%#` (#` (# XX 8.0  AUTHORSRECOMMENDATIONS..............................................................................31(#(#   8.10 ` Implementamorestructuredapproachtodefiningrequirementsforthedata  capturesystem........................................................................................................31r ` (#` (#  XX   XX 8.20 ` Researchthepotentialofanalldigitaldatacaptureenvironment.........................31# XX~(#^ ` (#` (# XX 9.0  CONCLUSION...............# XXK)#XX ...................................................................................................32\ (#(# 0   (#(# REFERENCES...............................# XXȴ)#XX ..............# XX*#XX .................................................................................33 4  # XX+#   &+v&% 6 6 6  Census2000DataCapture     1.0  Introduction <(#(# XX TheCensus2000Testing,Experimentation,andEvaluationsProgramprovidesmeasuresof  b effectivenessfortheCensus2000design,operations,systems,andprocessesandprovides  N informationonthevalueofnewordifferentmethodologies.AsacomponentofthisProgram,  : theobjectiveoftheCensus2000DataCaptureTopicReportistosynthesizethekeyfindingsof  & evaluationsandassessmentsthatpertaintovariousdatacapturetopics.Thereportidentifies    lessonslearnedduringCensus2000inordertohighlightissuesthatcanhelpguidepostCensus   2000planningresearchandtheupcomingdevelopmentalprogramsforCensus2010.Census   2000Testing,Experimentation,andEvaluationreportsareavailableontheCensusBureaus   Internetsiteat:4= <O  5  http://www.census.gov/pred/www/60<O0  7 0. r     `     h      p  XX2.0  Background 6(#(#    `    XX Thisreportprovidesanoverallsynthesisofissuesthatwereidentifiedinseveralstudies 6 addressingthetechnicalandoperationalelementsofthecomplexandlargescaleCensus2000 "r datacapturesystem.Xo XXXTheU.S.CensusBureauoutsourcedthetwomajorcomponentsofthe ^ Census2000datacaptureprogram.ThosecomponentsweretheDataCaptureSystem2000 J (DCS2000)whichwasawardedtoLockheedMartinandtheDataCaptureServicesContract 6 (DCSC)awardedtoTRW.LockheedMartinprovidedequipmentforimaging,recognition,and " datakeyingaswellastheprocessingsystemsforfourDataCaptureCenters(DCCs).TRW  providedstaffandservicesfordatacapture,facilitiesmanagement,officeequipment,supplies,  andofficeautomationforthreeoftheDCCs.(AfourthDCCwasmanagedbytheNational  ProcessingCenter(NPC),apermanentCensusBureaufacilityinJeffersonville,Indiana.)#XX XXow3# Xo XXXWithin  thereport,adistinctionismadebetweenthetwocomponents,asappropriate. n #XX X Xo6#TheunderlyingsystemtechnologywasdevelopedthroughacontractawardedtoLockheed F! Martin.Thecontractorcharacterizedthisprogramasoneofthelargestimageprocessing 2" projectsinhistory. ThedatacapturesystemXo XXXprocessedandcaptureddatafrom152million #n  censusformswithanextremelyhighaccuracyrate,whichexceededestablishedgoals(see  $Z! Section4.1).Inactuality,thetotalnumberofcensusformsexceededthisfigure.Basedona $F " cost/benefitanalysis,lowvolumeformsweredeliberatelyexcludedfromDCS2000asarisk %2!# mitigationstrategy.Theautomatedsystemwas,infact,designedtoprocess80percentofthe &"$ formsvolumewhiletheremaining20percentoflowvolumeformswereprocessedinadifferent ' #% manner. (#& Advancedtechnologieswereemployedtocaptureformsbycreatingadigitalimageofeachpage ~*%( andtheninterpretingrespondentsentriesusingOpticalMarkRecognition(OMR)andOptical j+&) CharacterRecognition(OCR)processes7  #  1      .#XX XXo8# Xo XXX??US.,ThiswasthefirsttimethattheCensusBureauhad  usedhighspeedOCRtechnologytocapturehandwrittenentriesbyrespondents.Although  OMRhadbeenusedin1990,theautomationin2000wasmoresophisticatedbecauseitincluded  keyfromimage(KFI)andOCRtechnologiesaswellasOMR.#XX X Xo<#US.,??.,=ԀThesystemwashighly t automatedbutstillreliedonextensiveoperationalsupportfromcontractors.Xo XXXDespitethereliance ` ontechnology,manualdataentrymethodswerestillneededtocapturedataincaseswherethe L  datawerenotmachinereadable,oriftheformwasdamagedandcouldnotbescannedorifthe 8  formswerelowvolume. $ t Oneaspectofthedatacapturesystemthathasperhapsbeenovershadowedbythehighlyvisible  L  useoftechnologyisthecontrolprocessesusedtomanagetheflowofformsthroughthedata  8  capturesystemandtomonitorimagequality.Thisworkflowmanagementsystemwasavery $  effectivemechanismthatensuredalldatawerecaptured.AccordingtoLockheedMartin,1.2   millionformswerererunthroughthesystem.Althoughthiswasasmallpercentageofthe   overallnumberofformsthatwereprocessed,itnonethelessprovidedanindicationofthe   stringentcontrolsappliedtomonitortheprocess.Therewasasignificantamountofcensusdata   associatedwiththose1.2millionforms.Differenttypesandsizesofformswereprocessed p throughDCS2000and,inadditiontocapturingrespondentanswers,DCS2000electronically \ interpretedidentificationandcontrolinformationontheforms.DCS2000hadanautomated H monitoringfeaturethatexaminedimagequalitybydetectingoveradozentypesoferrors.A 4 formrecoveryprocedurewasdevelopedandimplementedtohandlequestionnaireswiththose  p typesoferrors.  \ Thedatacapturesystememployedatwopassapproachtocapturedata.Thefirstpass 4 commencedonMarch6,2000andwascompletedonSeptember15,2000.Itcapturedthe100   percentcensusdata(fromboththelongandshortforms)neededforapportionment.Thesecond   passcapturedthesocialandeconomicdata(i.e.,thesampledata).#XX XXoF?#Xo XXXԀThiswasashorterphasethat  startedonAugust28,2000andcompletedonNovember15,2000.Thetwopassapproachwas  usedbecausetheoriginalkeyingrateestimatesweretoooptimisticandthetwopassapproach  wouldensurethatdatacapturedeadlinesweremet.TheaccuracyrateforOCRandOMRduring l bothpassesexceededprogramgoals.Themanualkeyingaccuracyratealsoexceeded X  expectations. D! LockheedMartin,theprimecontractorforDCS2000,citedthesystemicnatureofDCS2000 #l! whenexplaininghowitachievedhighaccuracyrates(LockheedMartin,2001b): $X" 0  Automateddatacaptureandthequalityoftheinformationproducedliesattheheartof %0!$ theDCS2000system.Manytimesintheimageprocessingindustry,productsorsystems &"% claimautomatedcharacterrecognitionratesof99%orhigher.Buttheseratesare  frequentlycalculatedonpreprocessedcharactertestdecksthatrarelygiveanindication  ofhowasystemwillworkinanoperationsenvironment.DCS2000canmakethesame  accuracyclaim,butataquestionlevelandonliveCensusproductiondata.Moreover, t thisrateisobtainedwithnearly80%ofthedatacapturedautomatically.Thislevelof ` automatedcapturedidnotcomefromsimplyacarefulselectionofcommercialproducts L  orevenbyfinetuningtheindividualOCRandOMRcomponents.Theseproduction 8  statisticsaretheresultofindepthtuningandcomplexintegrationofeverycomponentof $ t thesystem. `(#(# Indeed,therewere15commercialofftheshelf(COTS)productsintegratedintothesystem.  8  Thisapproachwasnecessarygiventhelimitedtimeavailabletodevelop,test,anddeploythe $  system.TheCOTScomponentsprovidedthefollowingfunctions:mailcheckinandsorting;   papertodigitalimageconversion;databasemanagement;workflowmanagement;digitalimage   processing;opticalcharacterandmarkrecognition;datareviewandcorrection;digitaltape   backupandrecovery;andsystemadministration.Theintegrationandtuningofthese   componentsweremajoraccomplishmentsgiventhecomplexityoftheDCS2000architecture. p #XX XXoG#AccordingtotheDataCaptureProgramMasterPlan(PMP)(BrinsonandFowler,2001),ofthe H approximately152.3millioncensusformsenteredintodatacapture,approximately83.9million 4 weremailbackforms,59.7millionwereenumeratorforms,600,000wereBeCountedforms,and  p 8.1millionwereGroupQuarters(GQ)forms8  #  2      .TheDataCapturePMPreportedthatacost  \ modelprojectedthatthetotalnumberofformstobeprocessedwouldbe149.7million.Itfurther H statedthatapproximately1.5billionformpageswereprocessedduringthedatacaptureperiod. 4 DCS2000outputfilesweretransmittedtotheXo XXXDecennialSystemsandContractsManagement   Office#XX XXoyU#Ԁ(DSCMO)onadailybasis.Inordertomanagethisenormousworkflow,DCS2000   continuallygeneratedprogressreportsformanagement.Xo XXX  Theoverallmanagementofthedatacapturesystemwasacriticalelementcontributingtothe  systemssuccess.InadditiontotheNPCandthethreeDCCs,anOperationsControlCenter l (OCC)wasestablishedinLanham,Marylandtooverseealldatacaptureoperations.Toassistthe X  OCCwiththemanagementoftheDCCsandtheirassociatedoperations,theDCSCManagement D! InformationSystem(DMIS)wasdevelopedtoprovideavarietyofintegratedofficeautomation 0"  tools.RawdataweretransmittedtotheDSCMOonadailybasis. #l! Thedatacapturesystemsucceededinprovidingthepopulationdataneededforpurposesof $D # determiningcongressionalapportionment,redistricting,andthedistributionofover$100billion %0!$ offederalfundstostateandlocalgovernments. &"% #XX XXoV# (#'   XX3.0  ScopeandLimitations XX (#(# Themainfocusofthisreportistoaddressthefollowingfourtopics:performanceofthedata  capturesystem;thesystemsabilitytocapturequestionnairedata;theimpactofdatacapture  requirementsonthequestionnairedesignandotherfactors;andtheappropriatenessof  requirementsidentifiedforthedatacapturesystem.Othersalientobservationsareincludedas t  wellinviewoftheirpotentialimportancetofuturedatacapturesystemsandprocesses.The `  followingdocumentswerereviewedforthisreport: L  0  1.DataCaptureProgramMasterPlan(PMP)!DataCaptureSystemsandOperations$ t (#(# 0  2.R.3.d.Census2000DataCaptureSystemRequirementsStudybyTitanSystems L  Corporation8 (#(# 0  3.K.1.b.EvaluationoftheQualityoftheDataCaptureSystemandtheImpactof   QuestionnaireCaptureandProcessingonDataQuality (#(# 0  4.LockheedMartinPhaseIILessonsLearned(includingAppendixA,Technical  LessonsLearnedWhitePaperp(#(# 0  5.TRWLessonsLearnedfromDCSCFinalReportH(#(# 0  6.RochesterInstituteofTechnologyResearchCorporationDCS2000DataQuality p(#(# 0  7.Census2000QuestionnaireDesignStudybyTitanSystemsCorporationH(#(# 0  8.AssessmentReportforDataCaptureofPaperQuestionnaires,preparedbyAndreaF.   BrinsonandCharlesF.Fowler,DecennialManagementDivision (#(# 0  9.LessonsLearnedforCensus2000,theFormsDesignandPrintingOffice(#(# 0  10.MemorandumfromHowardHogan,January24,2000.Subject:ProposalforQuality l! AssuranceofCensus2000DataCapture.X" (#(# 0  11.MemorandumfromDanielH.Weinberg,December7,2000.Subject:Actionsto 0$" CorrectPass2KeyingErrorsinCensusSampleMonetaryFields.%l #(#(# Onlytwoofthereferencesources(#3and#6above)arebasedonempiricalresearch.Allother &D"% sourcesprovidequalitativedata. '0#& Inaddressingthetopicsidentifiedabove,thisreportsummarizesthekeyfindingsandmajor )%( recommendationsofthedocumentsreviewedandseekstoidentifyanycommonthemesor *%) conflictinginformationbetweenthem.Therefore,thisreportisahighlevel,integrated +&* assessmentratherthanbeingacritiqueofeveryfacetofeachstudyreviewed.Itisnottheintent  ofthisreporttorevisitthedetailedstatisticaldatacontainedinthedocumentsthatwere  reviewed.  Limitationsstatedinotherreferencesourcesalsoindirectlyappliedtothisstudy.ThetwoTitan ` studiesandtheK.1.bevaluationcitedthelimitsidentifiedbelow.Specificdetailsoneachlimit L  aredefinedwithintherespectivedocumentsandarenotfullydescribedhereduetospace 8  limitations. $ t Census2000DataCaptureSystemRequirementsStudy  L   (Ax3"3"  3i2A3  0    Theperceptionofthosepersonsparticipatingintheinterviewprocesscansignificantly $  influencethequalityofinformationgathered3ii݌ (#(# Ќ  "3"  3k2A3  0    Insomecases,interviewswereconductedseveralmonths,evenyears,aftertheparticipant   hadbeeninvolvedinsystemdevelopmentactivities3k=k݌ (#(# Ќ  "3"  3hl2A3  0    Eachinterviewwascompletedwithinaonetotwohourperiod,withsometelephone \ followuptosolicitclarificationoninterviewresults3hll݌H(#(# Ќ  "3"  3m2A3  0    Everyeffortwasmadetoidentifykeypersonnelandoperationalcustomerswhoactively  p participatedindevelopmentefforts3mm݌ \(#(# Ќ  ! !Census2000QuestionnaireDesignStudy   "3"  3do2A3  0    Theperceptionofthosepersonsparticipatingintheinterviewprocesscansignificantly  influencethequalityofinformationgathered3doo݌(#(# Ќ  "3"  3p2A3  0    NearlytwoyearshavepassedsinceparticipantswerelastinvolvedinsupportingCensus l 2000activities3pp݌X (#(# Ќ  "3"  3q2A3  0    Duetoavailabilityproblems,Titananalystswereunabletointerviewthefullrangeof 0"  personnelwithknowledgeaboutprocessingissues3qr݌#l!(#(# Ќ  ! !K.1.bEvaluation $D # "3"  3s2A3  0    RawdataarenotarandomrepresentativesampleoftheU.S.population3ss݌&"%(#(# Ќ  "3"  3t2A3  0    Failuretoobtainalldataoriginallyplanned3tt݌(#'(#(# Ќ  "3"  3hu2A3  0    Resolutionof666,711recordsnotmatchedtothetwelveregionalcensuscenterfiles3huu݌|*%)(#(# Ќ   h+&* "3"  3v2A3  0    Subjectivityininterpretingthemostlikelyintentoftherespondent3vv݌(#(# Ќ  "3"  3w2A3  0    Datareflectmultiplesourcesoferrorbeyondthoseattributabletosystemdesign3ww݌(#(# Ќ  ! ! Thecollectionofdocumentsreviewedforthisreportidentifiedimportantissuesrelatedtodata ` capturetopics.Therewereadditionalevaluationsofdatacaptureoperationsplanned,whichmay L  haveidentifiedmoreissues.However,theseevaluationswereeithernotavailablebythetime 8  thisreportwascompletedorwerecancelledaltogether.Initially,thisreportintendedtoreflect $ t thecontentofupto11documents,butduetothesmallernumberofreferences,theconsolidated  ` findingsandrecommendationswillnotbeasextensiveasoriginallyplanned.Despitethis  L  limitation,thereportstillcoversabroadrangeofdatacaptureissues,reflectingbothquantitative  8  andqualitativeassessments. $  0   (#(#   XX4.0  DataCaptureTopicsAddressedinthisReportXX  (#(#  Anexpansionofeachtopicisprovidedbelowtogiveanappreciationforthescopeofissuesthat  wereexaminedacrossallofthedocumentsreviewed.   0  4.1Performanceofthedatacapturesystem \(#(# 0  Inordertoaddressperformanceissues,acleardefinitionforthesystemsobjectivemust 4 bearticulated.Thedatacapturesystemwascomprisedofbothautomatedandmanual  p processes.Datacaptureequipmentandrelatedsystemswereacquiredthroughacontract  \ awardedin1997toLockheedMartinMissionSystems.Theautomatedsystemscanneda H varietyofformsandcreateddigitalimagesthatwerereadbyOMRandOCRsoftware. 4 OMRaccomplishedmorethanmerelyidentifyingmarksinboxes.Itwascapableof   recognizingcaseswhenmultiplemarksappearedandusedanOpticalAnswer   Recognitionfeaturethatappliedanalgorithmandlogicprocesstodeterminethemost  likelyintendedresponse.TheOCRcomponentwasevenmoresophisticated.The  LockheedMartinstudynotedthatOCRaccuracywasafunctionofbothitsinherent   abilitytorecognizecharactersanditscontextualrecognitioncapabilities.Theexcerpt l! below(LockheedMartin,2001b)explainshowtheOCRengineachievedthehigh X"  accuracyrate:D#!(#(# 0  0` (#(#First,notonlydoesitrecognizecharacterswithahighdegreeofaccuracy,italso %l # providesmultiplechoicesforeachcharacterandcorrespondingbounding &X!$ charactercoordinates.Thisallowssubsequentcustomdevelopedcontextual &D"% processingtovalidatesegmentationresultsaswellasuseananalysisofmultiple '0#& recognitionhypothesesincontextandtheirprobabilitiesofoccurrenceinorderto ($' furtherimprovetheresults.Also,byprovidingadictionarylookupcapabilityas )%( wellasthedescriptionoftheprocessingusedtomatchorrejectawordasa *%) dictionaryentry,theproductallowsevenmoreopportunityfordownstream +&* analysisofthedataduringcontextualanalysis.Finally,becausetheproduct  providesavastarrayofdefinitionparameters,itisalsocustomizedtotreateach  individualfieldwithahighdegreeofdetailandspecificity,whichwillalso  maximizetheaccuracyandacceptanceratesoftheoutput.t` (#` (# 0  AnAutomatedImageQualityAssessment(AIQA)applicationanalyzedeachimaged L  document.Itcorrectedproblemsandenhancedimageswherepossible.Oncetheforms 8  wereconvertedintoanelectronicformat,theDCS2000softwareinterpretedthedataon $ t theformstothegreatestextentpossible.InthosecaseswhereOMR/OCRcouldnot  ` interpretthedatawithinacertainrangeofconfidencelimits,theformimagewas  L  automaticallysenttoKFI(keyfromimage),anoperationthatrequiredanoperatorto  8  interpretthe lowconfidenceresponsedataandthenmanuallykeythedataintothe $  system.Thus,astheRochesterInstituteofTechnologyResearchCorporation(RITRC)   putit, KFIgotthebulkofthemessyorambiguousresponses.TheKFIprocesswas   describedintheDataCapturePMPasfollows: (#(# 0  0` (#(#Theoperatorswerepresentedwithanimage,calleda snippet,oftheportionof p theformtheyweretokey.Ifafieldrequiredanaction,thecursorwaspositioned \ onthatfield.Usingtheirbestjudgement,theoperatorsthenkeyedallthe H charactersastheyunderstoodthemfromtheimage.Forseveralsituations, 4 keyingruleswereprovidedtoassisttheoperatorsininterpretingtheinformation  p andenteringstandardresponses. \` (#` (# 0  TheDataCapturePMPnotesthatfieldsreadbyOCRanddesignatedas lowconfidence 4 images,andthereforeautomaticallysenttoKFI,wereoftencorrect.KFIhaditsown   qualityassuranceprocessinvolvingcomparisonswithOCRand/orasecondkeyer.   FormsthatcouldnotbeimagedwererunthroughKFP(keyfrompaper)tocaptureall  datamanually.KFPinvolvedtwokeyers,withthesecondprovidingverificationofthe  dataenteredbythefirstoperator.(#(# 0  RITRCssamplingofproductiondatalookedattheacceptancerate9  #  3      ׀forbothOCRand X  OMRforthemailbackandenumeratorforms(100percentdata).FortheD1shortform D! andtheD2longform,theacceptanceratewas83.08percentforOCRand99.89percent 0"  forOMR.FortheD1EandD2Eenumeratorversionsoftheseformstheacceptance #l! ratesforOCRandOMRwereslightlylowerat79.17percentand99.78percent, $X" respectively.Basedonthesefindings,andotherconsiderations,RITRCconcludedthe $D # dataqualityfrombothsetsofformswasaboutthesame,withbothexceedingthe %0!$ programgoals.&"%(#(#  '#& Ї0  ThequalityofOMR,OCR,KFI,andKFPwasconstantlymonitored.Theaccuracyrates  forOMRandOCRdatacapturewerecontractuallyspecifiedas99percentand98  percent,respectively.(OCRaccuracywasactuallysubdividedintotwoseparate  accuracyratesof98percentforalphabeticdataand98.5percentfornumericdata.)Keyer t accuracyforKFIandKFPwasalsomeasured.TheaccuracystandardforKFIwas96.5 ` percentandKFPwastohavenomorethana2percenterrorrate.L (#(# 0  Giventhecomplexityofthedatacaptureenvironment,thevolumeofformsprocessed, $ t andtheuseofstateofthearttechnologies,itisinstructivetoexaminetheperformanceof  ` theoverallsystem.TheexaminationofDCS2000performanceissuesisnotintendedto  L  beafaultfindingexercise.Rather,itprovidesaviewintoissuesthatcanleadtoabetter  8  understandingoftheeffectivenessofthesystemusedduringCensus2000.This $  informationcan,inturn,benefitfuturedatacaptureoperationsbyprovidinginsightsinto   thebenefitsandlimitationsthetechnologyandmanualsystemsemployed. (#(# 0   4.2Thesystemsabilitytocapturequestionnairedata  (#(# 0  Asignificantdropinthenationwidemailresponserateduringthe1990Censusledto X dramaticchangesinquestionnairedesignstrategiesforCensus2000.Xo XXXThemajorimpetus D forchange #XX XXoؚ#Xo XXX inthequestionnaire#XX XXoM# Xo XXXdesign#XX X Xo#Xo XXXԀcameasaresultofCongressionaldirection,which 0 broughtabouteffortstomakethemailbackformsmorerespondentfriendly.The l assumptionwasthatrespondentfriendlyformswouldleadtoanincreaseinresponse X rates.#XX XXo#DuringthedecadeleadinguptoCensus2000,theCensusBureauconducted D researchintoformsdesignissuesinanefforttoincreasemailresponseratesandimprove 0 dataquality.Therewereanumberofmethodologicalteststargetedatimprovingthe  CensusBureau'sunderstandingofthearrayofcognitive,technical,andoveralldesign  factors.Thesefactorsinfluencetheefficiencyandaccuracyofdatacollectionandcapture  processes.Thetestingincludedarangeofstudiesthatexaminedthelayoutofthe  questions,thetestingofmatrixformatsagainstindividualizedformats,andevaluationof | differentenvelopecolors.  Xo XXX h(#(# 0  #XX X XoT#Ԛ Xo XXX??US.,Reflectingthecombinationofnewdesigninitiativesandtheavailabilityofsophisticated @! scanningtechnologies,theshortformunderwentsignificantchangesfortheCensus2000. ,"|  Theresultingformexhibitedanentirelynewandmorerespondentfriendlyindividual #h! spacelayout(separatepanelforeachperson)andprovidedfeaturessuchasabackground $T" color,motivationalicons,aCensus2000logo,checkboxes,andsegmentedwritein $@ # spaces.#XX X XoƠ#US.,??., ` %,!$(#(#  0  LockheedMartincametoappreciatethecriticalityofformsdesignanditscontributionto '#& capturingrespondentdata(LockheedMartin,2001b):(#'(#(# 0  0` (#(#Ofalltheaspectsofanautomateddatacapturesystem,theabsolutelymost x*%) criticalcomponentofthesystemisthedesignandprintingoftheforms.Agood d+&* formdesigncanincreasethestability,flexibility,errordetectionandrecovery,  andperformanceofthesystem.Apoorformdesigncanadverselyaffectallof  thesefactorsandsubsequentlyincreasethesystemcostexponentially.The  experiencesofDCS2000helpedtoemphasizethesepoints. t` (#` (#  0  TheAssessmentReportonDataCaptureofPaperQuestionnaires(BrinsonandFowler, L  2003)pointedoutsomeparticularformsdesignandsubsequenteffectsonprintingthat 8  affectedthesystemsabilitytocapturedata.Itobservedthat FormsDesignandPrinting $ t wasnotcoordinatedwiththedatacapturetechnology...untillaterintheprocessmakingit  ` moredifficulttodesignandtestthedatacapturetechnology.Thereportsuggestedthat  L  theautomationtechnologyavailablemaynothavebeenfullyutilized.Thereportfurther  8  statesthefollowingwithregardtohowcertainformsdesignandprintingissuesimpacted $  theOMRandOCRsubsystems: (#(# 0  0` (#(#Themultipleformtypes,bookletstyleformats,questiondesign,andspecific   colorsusedmadetheimplementationofOMRandOCRtechnologymore   challenging.Also,thelatenessinfinalizingquestionnairesandprintingof p prototypesmadethedevelopmentofOMRandOCRsoftwaremorecomplicated, \ ofhigherrisk,andmorecostly.H` (#` (# 0  Theneedforformsdesignandprintingtobetightlyintegratedwithintheoveralldata  p capturesystemdevelopmentenvironmentisapparentandwasechoedinseveralofthe  \ documentsreviewedforthisreport.H(#(# ̀ 0  4.3Theimpactofdatacapturerequirementsonthequestionnairedesignandother   factors Xo XXX??US.,  (#(# #XX X Xo#Xo XXXUS.,??.,ŭ0  Therewereseveralimagecapturespecificationsthatcreatedconstraintsintheforms  designenvironment.IntheCensus2000DataCaptureSystemRequirementsStudy,Titan  identifiedthefollowingareaswhereDCS2000hadclearlydefinedspecifications(Titan, l 2002):X (#(# "3"  0  3ɯ2A3  0` (#(#  SizeofScannableDocuments.Asetoffourspecificpapersizedimensionsfor 0"  singlesheetswasapprovedforDCS2000.Thebookletquestionnairesalsohad #l! definedlimitsforthesizeofseparatedsheets.AccordingtoDSCMO,DCS2000 $X" processedsixdifferentsizedforms,butwasnotlimitedtothisnumber.3ɯ݌$D #` (#` (# Ќ  "3"  0  3 2A3  0` (#(#  OpticalCharacterRecognition(OCR)WriteinFields.AsnotedbyRITRC,the &"% OCRsubsystemwasdesignedtoreadallofthewriteinfieldsforwhichtherewas '#& ahighlevelofconfidence.Consequently,therewereavarietyofveryprecise (#' criteriadefiningdimensionsandspacingrequirementsforthesefields.Thebasic )$( purposeofthesecriteriawastofacilitatecharacterrecognitionbythedatacapture |*%) system.3 D݌h+&*` (#` (# Ќ  "3"  0  3޴2A3  0` (#(#  OpticalMarkRecognition(OMR)CheckBoxes.Inadditiontosizeandspacing  requirementsfortheseboxes,therewerealsospecificationsindicatingthatboxes  ononesideofthepageshouldnotcoincidewithcheckboxesorimageareason  thereversesideofthepage.However,accordingtoDSCMO,paperspecifications t requiredahighopacityleveltominimize showthrough.3޴݌`` (#` (# Ќ  "3"  0  32A3  0` (#(#  ColorandScreen.Amajordatacapturerequirementwasthatthebackground 8  colormust dropoutandshouldnotcontainanyblackcontent.Adropouteffect $ t canusuallybeachievedthroughacombinationofthecolorandscreendotsize.3݌ `` (#` (# Ќ  "3"  0  3l2A3  0` (#(#  Margins.Requiredwhitespacewasdefinedforside,top,andbottommargins.3l݌ 8 ` (#` (# Ќ  "3"  0  32A3  0` (#(#  FormType/PageIdentifier.Therewasasetofrequirementsfortheuseofthe   Interleaved2of5barcode,whichservedtoidentifyformtypeandpagenumbers.3ú݌ ` (#` (# Ќ  "3"  0  32A3  0` (#(#  DocumentIntegrity.Sincebookletformscouldbecomeseparatedduringthe   scanningprocess,auniqueidentifierhadtobeincludedonallsheetsofalong p formtolinkthesheets.Anotherbarcodewasprintedinthemarginareato \ providethesheetlinkagefunctionnecessarytoensureDocumentIntegrity. H DocumentIntegritywasalsoincludedonbothsidesoftheshortformtomitigate 4 theriskofnondetecteddoublefeeds.3H݌ p` (#` (# Ќ  0  #XX XXo#??US., Xo XXXTheconstraintsimposedontheformsdesignbythedatacapturerequirementsmustbe H viewedintermsoftheircontributiontohighlyefficientdatacaptureprocesses.Inall,26 4 differentformswerescannedusingOMRandOCRtechnologies,resultingina   substantiallaborsavingsachievedfromDCS2000.#XX X Xo1#US.,??.,Ԁ Xo XXX??US., (#(# ̚#XX X Xo#US.,??., 0  4.4Theappropriatenessofrequirementsidentifiedforthedatacapturesystem (#(# 0  Likeallsystems,theCensus2000datacapturesystemwasdesignedtosatisfyasetof l requirements.Asystemcannotprovidetherightfunctionalityiftherequirementsdefined X  foritwereincomplete.Thus,theefficacyoftherequirementsdefinitionprocess D! determinestoagreatextenthowwellthesystemwillwork.0" (#(# 0  Researchintonewtechnologiesthatcouldmakethedatacaptureprocessmoreefficient $X" beganintheearly1990s.TheRochesterInstituteofTechnology(RIT)testedavarietyof $D # commercialofftheshelf(COTS)productsin1993and1995.ARequestforProposals %0!$ (RFP)wasdevelopedin1996toprocureproventechnologiesandtooutsourcethe &"% developmentandoperationofthedatacapturesystem.'#&(#(# 0  Inthepreawardphase,multiplevendorswereaskedtoconductanoperational )$( capabilitiesdemonstration.AccordingtotheCensus2000DataCaptureSystem |*%) RequirementsStudy,thisdemonstrationallowedtheCensusBureautoidentifythe h+&* contractormostsuitedtothetaskofdevelopingDCS2000andservedtoidentifyand  finetunerequirementsforthedatacapturesystem.TheawardtoLockheedMartinwas  issuedonMarch21,1997anddevelopmentactivitiesensued.(#(# 0  Theoriginalstatementofwork(SOW)wasusedfordevelopmentuptotheCensus2000 ` DressRehearsal.AtthatpointitwasdeterminedthattheSOWlackedsufficientdetail L  andrequiredmorespecifics.Consequently,aFunctionalBaseline(FBL)documentwas 8  developedtofocusonwhatfunctionalitywasneeded.TheFBLwasgiventoLockheed $ t Martin,buttherequirementsforDCS2000continuedtoevolvethroughoutthe  ` developmentofthesystem.Similarly,TRWhadtorefinetheOperationsandFacilities  L  Planasdevelopmentproceeded.Thebasicrequirementsforthisplanweregenerated  8  jointlybytheCensusBureauandLockheedMartin.Therequirementswerealsoincluded $  intheRFPforDCSC,butTRWwasnotawardedthecontractuntilnearlyayearafter   LockheedMartinhadcommenceddevelopmentofthesystem.Bythistime,refinements   totheplanneededtobemadebyTRWtokeeppacewithDCS2000developments. (#(# 0  Thequestionposedinrelationtorequirementscanbeputinthecontextofdefiningwhat p theentiredatacapturesystemneededtodonotjusttheautomatedscanningtechnology. \ Therefore,thisreportlooksatrequirementsissuesfromboththeautomatedand H operationalaspectsofthedatacapturesystem.4(#(#   XX5.0  Findings XX H(#(#  Thissectionprovidesmajorfindingsandkeyissuesthatwereechoedinthedocumentsreviewed H forthisreport.Theyarediscussedwithinthecontextofthefourmaintopicareasofthisreport; 4 assessingperformance,factorsaffectingthesystemsabilitytocapturedata,issuesrelatingtothe   impactofdatacapturerequirements,andexaminingtheappropriatenessofthoserequirements.   Ѐ   0  5.1Assessingtheperformanceofthedatacapturesystem (#(# ??US.,0  5.1.1DCS2000exceededperformancegoals.  AsnotedintheDCS2000DataQuality h! reportpreparedbyRITRC,Census2000wasthefirsttimethattheCensusBureauhad X"  usedcommerciallyavailablescannersandhighspeeddigitaldataprocessingtocapture D#! censusdata.Althoughnotacompletelydefinitiveassessmentofqualityassurance(QA) 0$" forDCS2000,theRITRCanalysisconcludedthat Theresultsweobtainedfrom %l # ProductionDataQualitySamplingindicatethatDCS2000productiondataquality &X!$ significantlyexceededprogramgoals.US.,??.,;Xo XXX??US.,Puttingthissuccessintocontext,thedatacapture &D"% evaluation(Conklin,2003)notedthat althoughtheautomatedtechnologybrought '0#& increasedspeedandefficiencytoCensus2000processing,considerablehumanresources ($' werestillrequiredtohandlethemanymillionsofwriteinfieldsthatposedaproblemfor )%( it.#XX XXo#US.,??.,;*%)(#(#  +&* 0  Performancegoalsforaccuracywereexceededforbothmailbackformsandthe  enumeratorforms.RITRCstatedthat,basedonasamplingofimagesanddata,thein  production OCRaccuracywas99.6percent,KFIwasinexcessof97.8percent,andthe  checkboxquestionaccuracywasalittleover99.8percent.RITRCalsoreported t anothersignificantfinding:theoverallwriteinfieldaccuracyforbothpassesofdata ` capture(mergeddatacombiningOCRandKFI)was99.3percent.ForTitle13data L  (mergedfromOCRandKFI)onthemailbackforms,numericfieldswerefoundtohave 8  lowererrorratesthanalphafields.Thedatacaptureevaluationfoundthatforallfields, $ t OCRhadthelowesterrorrate,followedbyOMRandthenKFI. `(#(# 0  (Note:TheK.1.bfindingsconcerningerrorratesforthethreedifferentdatacapture  8  modessuggestadiscrepancywithrespecttoRITRCsfindings.TheOMRaccuracyrate $  reportedbyRITRCindicatesthatthismodehadthelowesterrorrate,howevertheK.1.b   evaluationplacedOMRafterOCRintermsoftheirrespectiveerrorrates(i.e.,OCRwas   lowerthanOMR).Thisapparentdiscrepancyiscomplicatedbythefactthatthesestudies   useddifferentperformanceclassificationmethods.Itisimportanttonotethaterrorrates   differfromaccuracyrates.Theerrorrateiscalculatedbysubtractingtheaccuracyrate p from100,typicallyyieldingaverysmallnumber(e.g.,.4percent),whereasaccuracyrates \ arethefractionofacceptedfieldsthatarecorrect,andarethereforeassociatedwithmuch H highernumbers(e.g.,99.6percent).Also,thestudieswerebasedondifferentdatasets. 4 AttemptingtocompareOCRtoOMRtoseewhichisthebestisnotavalidcomparison  p sincethetwomethodsaredesignedtoaccomplishdifferenttypesofrecognitiontasks.  \ GiventheexceptionallyhighperformanceoftheOCRandOMRsubsystems,bothof H whichexceededprogramgoals,thereisnoutilityinattemptingtodeterminea winner.) 4  `   (#(# 0  Thedatacaptureevaluationexaminedtheissueofwhetherornotsomefieldsweresentto   KFImoreoftenthanothers.Itconcludedthatnamerelatedfieldsweremorelikelytogo  toKFIunnecessarily.Italsoposedthequestionofwhethercertainfieldssent  automaticallytoKFIshouldbeprocessedinsteadbytheautomatedtechnology.The  discussionofthisissuesuggestsfurtherresearchisneeded:l(#(# 0  0` (#(#WenotesomefieldsautomaticallywenttoKFIregardlessofhowwellthe D! technologythoughtitcouldprocessthem.Thesewerecheckboxfieldswhere 0"  morethanoneboxcouldbeselectedandstillcountasavalidresponse. #l! RecognizingthatKFIissubjecttoerrorfromfactorsnotaffectingthetechnology, $X" e.g.,humanfatigueandinattention,apossiblefuturetestfortheautomated $D # technologyistoallowittoprocessmultipleresponsecheckboxfields.Itwould %0!$ behelpfultofindoutifthetechnologycanbeadjustedtoacceptsuchfields &"% withouttheerrorsofkeying.'#&` (#` (#  h+&* 0  OneimportantlessonlearnedcitedbyLockheedMartinduringtheCensus2000Dress  Rehearsalwasthatthe fullfieldmethod:  #  4      ׀ofkeyingprovidedthemostcosteffective  combinationofaccuracyandthroughputgiventherequirementsoftheDCS2000  system.Othertypesofkeyingmethodsmayhaveincreasedthroughput,thoughaccuracy t wouldhavesuffered.`(#(# ̀0  5.1.2Reasonsforerrors.AccordingtoRITRC,somepercentageofKFI errorsintheir 8  productiondataqualitysamplingwasduetoambiguoushandwriting,misinterpretationof $ t keyingrules,orvaryinginterpretationsofhandwrittenentries(whererespondentintent  ` wasnotclear).Therefore,thesewerenotconsideredasactualkeyingerrors.Thedata  L  captureevaluationnotedthattheKFIerroris notnecessarilyapoorreflectiononthe  8  automatedtechnologyandobservedthattheautomatedtechnologyandtheevaluation $  andproductionKFIarepronetothefollowingerrors: (#(# 0,XX XX00  0(#(#"3"  32A3  0 (#(#  failuretoreadafieldontheform3*݌  (# (# Ќ  0  0(#(#"3"  32A3  0 (#(#  pickingupcontentthatisnotreallythere3,݌p (# (# Ќ  0  0(#(#"3 "  3 2A3  0 (#(#  incorrectlycapturingthecontentonthepaper3 6݌H (# (# Ќ  0  0(#(#"3 "  32A3  0 (#(#  correctlycapturingwhattherespondentwrote,butnotnecessarilywhatthe  p respondentintended3C݌ \ (# (# Ќ  0  Theerrorratecanbeattributedtofactorssuchasthehardwareoftheautomated 4 technologyorthesoftware.LockheedMartinidentifiedthetwomaincontributorsto   systemerrors:noise(includingnoisegeneratedbyrespondents)thatinterfereswith   characterrecognitionandsegmentationerrors(multiplecharacterswithinoneboxor  singlecharactersspanningmultipleboxes).Thiswascommontootherimageprocessing  systemsaswellasDCS2000.(#(# 0  Anothermajorcauseofkeyingerrorswastheuseofcertainkeyingrules.Forexample, X  therulescalledforawriteinfieldtobefilledwith eightsifthenumbercouldnotbe D! determinedand ninesweretobeusedifavalueexceededcertainlimits.RITRC 0"  expressedconcernaboutkeyerconfusionovertheuseof eightsand nines, #l! particularlywithrespecttotheincomefieldwithitsimbeddeddollarsign,comma,and $X" preprintedcents.Theyalsonotedthat theessentialKFIproblem...wastheconflict $D # between keywhatyouseeandinterpretationsrequiredbythekeyingrules.The %0!$ DecennialStatisticalStudiesDivision(DSSD)hasalsoreportedproblemswiththe &"% incomefield,citingan errorrateofnearly45percentforincomefieldsfilledwith9s '#& and mistakes...infieldscontainingall8s.(Weinberg,2000).(#'(#(# Ї0  Thedatacaptureevaluationreported,acrossvariousmodesofdatacapture,thatthemost  frequentreasonsforfailingtocapturetheintendedresponseswere:(#(#   "3 "  0  32A3  0(#(#  Extracheckbox!theoutputfromtheautomatedtechnologyoutputshowsmore t checkboxesmarkedthanareinthescannedimage.3݌`(#(# Ќ  "3 "  0  3B2A3  0(#(#  Missingcharacters!theoutputfromtheautomatedtechnologyhasfewercharacters 8  thanthescannedimage.3B{݌$ t(#(# Ќ  "3 "  0  32A3  0(#(#  Wrongcharacter!theoutputfromtheautomatedtechnologyandthescannedimage  L  havethesamenumberofcharacters,butoutputfromthetechnologydisagreeswith  8  theimageinoneormorecharacters.3݌$ (#(# Ќ  0  Thesamereportlistedthemostcommonreasonsfortheseproblemsas: (#(# "3"  0  32A3  0(#(#  Poorhandwriting!therespondentshandwritingmakesoneletterlooklikeanother,   butapersoncantellwhattherespondentmeant.3݌p(#(# Ќ  "3"  0  3%2A3  0(#(#  Noreasonfound!theresponseiswrittenclearly,andthereisnothingtosuggest H whyitwasnotcapturedcorrectly.3%^݌4(#(# Ќ  0  Thedatacaptureevaluationreachedsomesignificantconclusions.First,itfoundthat if  \ thereisintelligentcontentinafield,theautomatedtechnologywilldetectitwithnearly H perfectcertainty.Second,despitethefactthatthesystemisnotperfect, asizeable 4 portionofresponseswillbecapturedandinterpretedcorrectlyatspeedsthatareordersof   magnitudeaboveKFI.Andthird, thelargestimpedimenttoautomationisnotthe   qualityofthehardwareorsoftware,butthequalityoftheresponsessuppliedbyhuman  beings.Thestudysuggeststhatattemptingtobuildasystemthatcouldcapturenearly  anytypeofresponsewouldnotbeapracticalendeavor.(#(# 0  5.1.3OCRacceptancerate.Accordingtothedatacaptureevaluation,althoughthe X  automatedtechnologybroughtincreasedspeedandefficiencytoCensus2000processing, D! considerablehumanresourceswerestillrequiredtohandlethemanymillionsofwritein 0"  fieldsthatposedproblems.ThepercentacceptedbyOCRforwriteinfields(shortand #l! longforms)was78.9percent,whichisquiteclosetothe81.2percentreportedbyRITRC $X" forPass1.However,RITRCreportedamuchlowerrateforPass2of64.8percent.The $D # lowerratereflectstheproblemsinherentininterpretingthemoredifficultlongform %0!$ responses.&"%(#(# 0  TheOCRacceptancerateforformscompletedbyenumeratorswaslowerwhencompared (#' totherateformailbackforms.RITRCbelievesthatthismayhavebeenduetolight )$( pencilmarksorincompleteerasures.Thus,thisloweracceptanceratewasprobablymore |*%) ofafunctionassociatedwiththewritinginstrument(i.e.,pencil),andnotareflectionon h+&* theeffectivenessoftheOCRsubsystem.(#(# Xo XXX0  5.1.4Clusterconcept.DCS2000wasbasedontheconceptofclusterprocessing.  Clusterswereautonomousunitsofimageprocessing,constructedaroundthecapacityof t threescanners.EachDCCwasequippedwithasmanyclustersasnecessarytoprocess ` theirworkload.InLockheedMartinsopinion,theclusterdesignwasakeyfactorthat L  contributedtothesuccessfuldevelopmentoftheoverallsystem.Inthisconcept,each 8  clusterfunctionedindependently,usingasetofmodules,withasequentialprocessing $ t orderthatverifiedthepreviousactionstakenbyothermodules.LockheedMartin  ` describedtheoperationoftheclusterconceptasfollows: L (#(# 0  0(#(#Thisefficiency[ofverifyingeachstepoftheprocess],whichfrequentlyexceeds $  automatedacceptanceratesof80%ofthedataatgreaterthan99%accuracy,also   incorporatestheselfvalidationthemesofclusterprocessing.Whiletheobvious   validationstepsarethekeyingoperatorfunctions,thereisalsosignificantuseof   crosscharacter,crossfield,andcrossimagecontextualchecksthatareperformedin   ordertovalidateandrevalidatethedatathatisproduced.Thisprocessingisalso p spreadacrossthevariousstepsoftheclusterworkflow.Atthecontextstep,OMR \ box,character,field,andimagevalidationsareperformedtomaximizetheefficiency H oftheautomatedOCRandOMR.Then,anothermoduleassuresthatcontroldata 4 andformintegrityhavenotbeencompromised.Next,keyfromimagefunctions  p (KFI)arethenusedtocompletethecaptureofdatathatcouldnotbedone  \ automatically.Whilethesekeyingstepsalsoincluderealtimeedits,functionsfor H operatorstorejectentireimagesforquality,andautomatedqualityassurance 4 checks,therearealsosubsequentfullformvalidationfunctionsthatassurethatthe   captureddataisconsistentwithexpectedresponses.Alltheseprocesseswork   togethertocontinuouslyvalidateandimprovethecaptureddataasitprogresses  throughthesystem.(#(# 0  Inshort,theclusterconceptensuredimagesweresubjecttostringentqualitychecks.The l primeDCSCcontractor,TRW,notedthatqualitywasdesignedintotheprocessduetothe X  shortterm,highvolumenatureofthework.Theyconcludedthat thisprovidedahigh D! qualityproductwithaminimumofQAstaffandoverhead.Inthisregard,theprime 0"  DCS2000contractor,LockheedMartin,statedonlyonepercentoftheformsneededtobe #l! pulledandreprocessed.Anotherbenefitoftheclusterarchitecturewasthatitallowedfor $X" continuedscanningofforms,evenwhentherewerecomponentfailureswithinacluster.#XX XXo!#$D #(#(#   0  5.1.5InefficienciesintheKFPprocess.TheDMDAssessmentReportonDataCapture &"% ofPaperQuestionnairescharacterizedKFPasbeingan inefficientwaytocaptureforms '#& thatcouldnotbecapturedbyscanning.ThereportnotedthatthesoftwareusedforKFP (#' wasdesignedforimagekeyingandwastherefore cumbersome.Italsotookissuewith )$( theKFPpolicyofrequiring100percentverificationstatingthatthismayhave caused |*%) morekeyingthanwasrequiredsinceasampleverificationmayhavebeensufficient.In h+&* OY    general,theDMDreportfavoredmaximizingtheuseofautomationandrelyinglesson  keying.(#(#       5.2Factorsaffectingthesystemsabilitytocapturequestionnairedata t  0  Notingthatmanyformsprocessingsystemscandefineaformtorunthroughthesystem H  inminutes,LockheedMartinobservedthatthecompletedefinitionofDCS2000was 4  dependentuponutilizingalloftheoptimizationsdesignedintotheformitself.This  p meanstheformultimatelyreflectedanindepthanalysisofthetotalenvironment  \ includingCensusrules,formcharacteristics,andrespondenttendencies.  H (#(# 0  5.2.1Keyingrulesandmethods.Whileallowingforrespondenttendencieswasfactored   intotheformsdesignprocess,therewasstillaneedtoapplykeyingrulesandmethods   forcapturingdata.AccordingtoRITRC,oneofthemajorcausesofkeyingerrorswas   thecomplexityoftherules,whichrequiredkeyerstomakeinterpretations!while   maintainingahighpaceofproduction.AccordingtoTRW,theapplicationofkeying   rulesvariedbetweenPasses1and2.Theynotedthattheapplicationofruleswaslimited l onPass1becauseofthelimitednumberoffields;therulesweremorecriticalinPass2 X becauseofthebroaderrangeoffieldtypes.TRWreportedthatdailyauditsshoweda D highdegreeofaccuracyduringPass2keying.Nonetheless,inviewoftheneedfor 0 interpretationsbykeyersandvariationsofrulesbetweenthepasses,themoreaccurately l theformsarefilledout,thelessneedthereisforkeying.Basically,thesystemsabilityto X capturequestionnairedatacanbeimprovedbybetterformsdesignpractices.D(#(#         h   0   5.2.2Improvementstoformsdesign.GiventhesuccessofDCS2000asahighspeed,  highvolume,highquality,datacapturemechanism,someaspectsoftheformcouldbe  improved.Thedatacaptureevaluationcitedandendorsedpossibleimprovementstothe  formthathadbeenidentifiedinTitansCensus2000QuestionnaireDesignStudy.(#(# 0  Thedatacaptureevaluationrecommendedthefollowingbeconsideredaspossible h improvements:T (#(# "3"  0  3"2A3  0(#(#  Havethepersoninformationforhouseholdmembersbefilledoutfromlefttoright ,"|  acrossthepageinsteadofupanddown.3""݌#h!(#(# Ќ  "3"  0  3#2A3  0(#(#  Allowtheuseofpencilssorespondentscaneasilycorrectmistakes.3#$݌$@ #(#(# Ќ  "3"  0  3$2A3  0(#(#  Changethesizes,fonts,appearance,etc.oftheinstructioniconssotheyareeasierto &"% spot(orsimplyeliminatethem).3$)%݌'#&(#(# Ќ  "3"  0  3H&2A3  0(#(#  Allowmorespacesforthelastnamefields.3H&&݌)$((#(# Ќ  "3"  0  3>'2A3  0(#(#  Includeinstructionsforfillingoutorcorrectingwriteinfields.3>'w'݌d+&*(#(# Ќ  "3"  0  3K(2A3  0(#(#  Includemoredetailedinstructionsfortheraceandethnicityquestions.While  additionalinstructionsmayimproverecognition,DSCMOandothers(e.g.,Dillman)  expressedconcernsthatanovercrowdedformwithtoomanyinstructionsmayhinder  responseanddatacapture.3K((݌t(#(# Ќ  "3"  0  3`*2A3  0(#(#  Trytomaketheinstructionstotheheadofhouseholdforfillingouttheformmore L  concise.3`**݌8 (#(# Ќ  "3"  0  3+2A3  0(#(#  EmploytheuseofheaderstoseparatetheAsianethnicityoptionsfromtheonesfor  ` PacificIslander.3++݌ L (#(# Ќ  "3"  0  3,2A3  0(#(#  Donotspreadthechoicesforcheckboxfieldsovermorethanoneroworcolumnon $  apage.3,-݌ (#(# Ќ  "3"  0  3.2A3  0(#(#  Selectabackgroundcolorwithbettervisualcontrast.3.Q.݌ (#(# Ќ  0  Enhancementstotheseareashavethepotentialtofurtherimprovethequalityofdata p capturedandperhapsmaketheformevenfriendliertorespondents.\(#(# 0  0(#(#(Note:TheCensus2000QuestionnaireDesignStudywasaqualitativestudythat 4 reflectedtheinsightsandexperienceofsubjectmatterexpertswhohadextensive  p knowledgeofformsdesign.Whilethefindingsfromthisstudysuggestedthatcertain  \ aspectsofthequestionnairecouldbeimproved,wecautionthatfurtherresearchand H testingisneededtodeterminewhich,ifany,oftherecommendationsshouldbe 4 implemented.Itisimportanttonotethatoneofthemainpurposesofthisstudywas   toidentifyandhighlightformsdesignissuesthatcouldbecandidatesforfuture   researchefforts.)(#(# 0  5.2.3Fewinstructionswereprovidedtorespondents.Severaloftheabovebulletstouch  onthissubject.Thesystemsabilitytocapturerespondentdatacouldhavebeen l impacted,toacertainextent,byalackofinstructionstorespondents.Asnotedinthe X  Census2000QuestionnaireDesignStudy,theshortformdoesnotprovideguidanceto D! respondentsonwhattodoifananswerexceedsthelengthofawriteinboxorhowto 0"  correctanymistakesontheform.Thestudysuggestedthatsomequestionscouldbenefit #l! fromexpandedinstructions,althoughtherearerisk/benefittradeoffsthatwouldneedto $X" beassessed.Itisconceivablethatadditionalinstructionsforrespondentscouldhave $D # reducedproblemsandenabledthesystemtocapturemorerespondentdata,ratherthan %0!$ rejectingitasunreadable.&"%(#(# Ѐ   '#&   Xo XXX??US.,0   5.2.4Useofbankingshouldbeminimized.Mostintervieweeswhoparticipatedinthe (#' Census2000QuestionnaireDesignStudyunderstoodtheneedforbankingbutfeltthatit )$( wasnotadesirabledesignfeatureanditsuseshouldbeminimized.Thistechnique, |*%) especiallytriplebanking,#XX X XoL7#Xo XXXUS.,??.,k7??US.,canleadtotightlygroupedcheckboxes.Besidesbeing h+&* confusing,itincreasedthelikelihoodofaprocessingproblemwhenalargemarkextends  beyondtheboxboundaryandintoadjacentboxes.#XX XXo9#US.,??.,I9ԀGreaterspacingbetweenboxesmay  minimizestraymarksthatcreateinterpretationproblemsforOMR.(#(#   0  5.2.5Theracequestion.Duetothemultipleanswersallowedbythisquestion,Lockheed ` Martinfoundthisquestiondifficulttoarbitratewithanydegreeofhighaccuracy.Owing L  tothehighimportanceplacedonaccuratelycapturingracedata,whenmultiplemarks 8  weredetectedbytheOMRsubsystem,theywerepassedtokeyersforresolution. $ t AccordingtoLockheedMartin,theaccuracyratesforthis particularlysensitiveareaof  ` interestincreasedsignificantlyasaresultofthemanualinterpretation. L (#(#  0  5.3Issuesrelatingtotheimpactofdatacapturerequirementsonthequestionnaire $  designandotherfactors  (#(# Xo XXX??US.,0  ThecompleteredesignofthequestionnaireforCensus2000producedamorerespondent   friendlyformbasedonanindividualspaceformat.#XX XXo|># Xo XXXԀ#XX X Xoy?#Xo XXXToalargeextent,thenewdesignof |  theformwasmadepossibledueto#XX XXo?# Xo XXXtechnologicaladvancementsmadeineԀ#XX X XoY@#Xo XXX@OCRandOMR h technologies.#XX XXo@#US.,??.,>T(#(# 0   (#(# 0  ??US.,Xo XXX5.3.1Spaceseparationbetweenboxesandtext.#XX XXoA#US.,??.,AXo XXXTherehadtobeaspaceseparation ,| betweenboxesandtextofatleast.08inches,andspacebetweenboxes(fromthebottom h ofoneboxtothetopofthenextbox)ofatleast.05inches.TheDCS2000imagecapture T specificationsnotedthatmorespacebetweenboxeswaspreferableinordertopreventa @ largemarkfromrunningintoanotherbox.Althoughconformingtothesespecifications, , manyboxesontheformweretightlygrouped.Forexample,theHispanicoriginandrace  questions,andquestion#2forPersons26containednumerousboxesthatweretightly  groupedwithminimalverticalspaceseparation.Thisincreasedthelikelihoodofadata  captureerrorwhenalargemarkextendedbeyondtheboxboundaryintoanadjacent  checkboxorsegmentedboxes.Futureuseofcheckboxesshouldallowgreaterspacing x asrequestedintheDCS2000imagecapturespecifications.#XX XXocB#d(#(# 0  5.3.2Darkerframearoundsegmentedboxesanddarkersegments.TheDCS2000 % #(#(# 0  TheAssessmentReportonDataCaptureofPaperQuestionnaireswasingeneral 'f"% agreementwiththeneedforaunifiedsystemdevelopmentenvironment.Itrecommended(R#&(#(# 0  integrateddevelopmentinvolvinginternalstakeholdersearlyintheplanningphaseand (>$' theneedfor betterintegrationofformsdesignandprintingspecificationswithdata )*%( capturesystemdevelopment.Itfoundthatformsdesignwaslargelyindependentofdata *&) captureandprocessingsystemdesignsandthereforethiswasafactorthatledto +'* questionnairedesignersbeingoverconfidentinthecapabilitiesofOMRandOCR  technologies.(#(# 0   7.20(#(#Definerequirementsearly. t(#(# 0  AsnotedinTitansCensus2000DataCaptureSystemRequirementsStudy,requirements H  establishtheveryfoundationforasystem.Theirimportancecannotbeoverstated, 4  especiallyinanenvironmentwhereasubstantialR&Dinvestmentisnecessary.Delaysin  p definingrequirements,ornotfullydefiningthem,increasesthelikelihoodthatthesystem  \ willnotmeetdatacaptureexpectationsorperformatthelevelrequired.Or,inthecaseof  H  qualityassurancerequirements,waitinguntillateinthedevelopmentcyclemaynotallow  4  forsufficienttimeforimplementationofthemechanismsneededtogenerateappropriate   metricsorensureadequatequality.Startingtheplanninganddevelopmentearlierwould   provideagreaterchancethatallidentifiedrequirementswillbeimplementedandthat   sufficienttimewillexistfortestingandsystemrefinement. (#(#      7.30  DevelopQAstandardsearly .l(#(# 0  GiventhatrequirementsforQAwerenotwellestablished,therecommendationisthat D moreemphasisbeplacedondefiningQAmeasurements,processes,andreportsforthe 0 2010Census.TRWrecommendedthatboththeDCS2000andDCSCparticipants l developanintegratedQAprogram.Additionally,asstatedintheRITRCreport,the X CensusBureaushouldinvestigatenewQAtechnologiestobringQAevaluationtime D closertoproductiontime.Inthisregard,theAssessmentReportonDataCaptureof 0 PaperQuestionnairespointedouttheneedforthedatacapturesystemtoprovideforreal  timeaccesstodataforqualityassurancereview.(#(#   7.40  Focusonredesigningkeydatacaptureproblemareas .(#(# Xo XXX0  Asdiscussedinthisreport,segmentationerrorsaccountformanyoftheprocessing h problems.Whileproblematic,thisfacetofformsdesignprovidesapotentiallyhigh T  paybackareaforfutureresearch.#XX XXo# Xo XXXDCS2000imagecapturespecificationsprohibitedthe @! useofdarkoutlinessurroundingOCRfields,buttheuseofadifferentbackgroundcolor ,"|  withahighercontrasttowhiteshouldalleviatethedefinitionproblem.#XX X Xo(##h!(#(#  0  AnotherexampleofasignificantproblemareaontheformsistheHispanicandrace $@ # question.ManyoftheparticipantsintheCensus2000QuestionnaireDesignStudy %,!$ agreedthatthiswasanespeciallytroublesomearea.Althoughtherewereimprovements &"% intermsofthepresentationandsequencingofthesequestionsin2000,thereisstillroom '#& forfurtherimprovementsthatcouldmakethemeasiertounderstandandlessproneto (#' interpretationproblemsduringthedatacaptureprocess.Moreempiricalresearchis )$( neededonthisparticulardesigntopic.x*%)(#(#   Xo XXX d+&* 0   7.5Limitthenumberofforms. 󀀀    (#(# 0  RITRCobservedthatthenumberofformsgrewto75forCensus2000(only26weredata  capturedbyDCS2000).Theyexpressedconcernsthat,withseparateformdefinition t templatesbeingcreatedforeachformandthateachhadtobetested,thisproliferationof ` formsbecame unwieldy.RITRChassuggestedthatthe 8020ruleL  #  6      ׀mightapplyin L  regardstothenumberofformsthatareusedforthe2010Census.Thatis,theuseof 8  fewer,moregenericformsmightprovetobecosteffectivefromadatacapture $ t perspective. `(#(#  0  Therearetwomajorissuestoconsider.First,thelevelofeffortrequiredtodevelopthe  8  templateshasmajortimeandcostimplicationsfortheCensusBureau.Second,since $  identicalinformationisbeingaskedacrossmultipleforms,thereisariskthatwordingor   phraseologymaynotbeconsistentacrossallforms.Therecommendationisforthe   CensusBureautoconsidercombiningforms,whenpossible,soasingleformmayserve   morethanonepurpose.Thereisariskinherentinoverrelianceontoomanygeneric   forms.p(#(# 0  TRWnotedthatitencountereddisruptionstooperationsduetotheneedtoadaptto H differenttypesofforms.TheyalsorecommendedtheCensusBureaushouldminimize 4 thenumberofformtypesanddesignthemasearlyaspossiblesotheprocessing  p implicationscanbeunderstoodpriortotheDCSCdevelopingitsprocedures. \(#(# 0   7.6Assessfutureroleofautomateddatacapturetechnology. 4(#(# 0  Thedatacaptureevaluationcitedtwopossibilitieswithregardtothefutureuseof   automateddatacaptureandimagingtechnologyinthedecennialcensus.Ifitisseenas  havingasupportingrole,itwouldbeusedprimarilyforrapidlycapturingtheclearand  easyresponses.Inthisscenario,traditionalmethods,althoughresourceintensive,would  stillbeusedtocaptureespeciallydifficultresponses.Ontheotherhand,automated l technologycouldhaveadominantroleassumingthatcensusformsweredramatically X  streamlinedandthelongformwithitsnumerouswriteinresponseswasnolonger D! capturedinthedecennialcensus.Ifthelongformisretained,thedatacaptureevaluation 0"  askedifitisthenworthwhiletoputahighpriorityonimprovingthequalityperformance #l! oftheautomatedtechnology.Thedatacaptureevaluationsuggestedthatthisissuecould $X" bearesearchquestionfortestingleadinguptothe2006CensusTest.$D #(#(# 0  Withpoorhandwritingaccountingformanyerrors,thedatacaptureevaluationsuggested &"% givingconsiderationtoreducingsomewriteinfieldstocheckboxes,reducingthesetof '#& questions,orusingmoreenumeratorstogetlongformdata.(#'(#(# Ї0  7.7ImplementaunifiedHelpDesk. (#(# 0   (#(# 0  InTRWsopinion,theexistenceoftwodifferenttypesofhelpdesks(i.e.,oneforDCS  2000andoneforDCSC) introducedunnecessarycomplexity,overlapandconfusion. p Theyrecommendusingonlyoneintegratedhelpdesk.\(#(#   0  7.8Trainontheactualworkingsystem.4 (#(#  0  Giventhesizeoftheworkforcefordatacaptureoperations(over10,000temporary  X employeesduringthedatacaptureoperationsphase),TRWwasconcernedwhetherall  D  employeeshadthenecessaryskills.TRWrecognizedtheneedforconsistencyintraining  0  acrossallsites.Toprovidequalitytraining,TRWcitedakeylessonlearned: Fromthe   startoftheproject,instructionaldesignteamsneedasmuchaccesstotheactualworking   systemandequipmentaspossibletoensureaccuratetrainingdocumentation.This   includesparticipationindressrehearsalsandOTDRs. (#(#  0  7.9Expandthetimefortesting. h(#(#         h   0  Aspreviouslymentioned,thevarioustestswereextremelybeneficialandexercisingthe < systemprovidedinvaluableinsights.However,TRWwasconcernedthattheOTDRs (x startedtoolateintheCensusschedule(August1999)andwouldliketoseemoretime d allottedbetweentests.SincemanagersfromallfourDCCsparticipatedineach P sequentiallyscheduledOTDR,theirtimewasdividedbetweenparticipationinanOTDR < atanothersite,preparingtheirownOTDR,andpreparingtheirsitefordatacapture ( operations.TRWnotedthat preparingforanOTDRisafulltimeprojectthatcould  havebeenperformedbetteriftheparticipantswerenotbusyatprevioustestsatthesame  timethattheyneededtobepreparingfortheirowntest.(#(#   0  7.10BetterdefineDMISforfuturecontracts. t(#(# 0  TRWexpressedconcernsaboutthelevelofeffortrequiredtoimplementtheDCSC L  ManagementInformationSystem(DMIS)andthefactthatitrequiredmoreresources 8! thananticipated.Theirassessmentwasstatedasfollows:$"t (#(# 0  0(#(#ThescopeofDMISwasseverelyunderestimated.InsteadofafewCOTStools #L" sittingondesktops,itgrewintoamediumsizednetworkedinformationsystem.This $8 # wasadifficulttasktocompleteinlessthantwoyears.Weshouldhavescaledback %$!$ thescopeofDMISgiventhetimeconstraint.Wewereunderstaffedearlyinthe &"% process.Itwasdifficulttoattractdeveloperstotheprojectduetothelackof '"& programmingworkandtheshorttermnatureofthisproject.(#'(#(# 0  Theseconcernsshouldbenotedforfutureplanningpurposesandthescopeof p*%) requirementsforsystemslikeDMISmustbefullyunderstood.Amanagement \+&* informationsystemtypicallyinterfaceswithnumerousothersystemsandthisincreases  thecomplexityofdevelopmentandtesting.Themagnitudeofeffortandthetime  requiredtodeveloprobustinformationsystemsessentialtothemanagementoflarge  scaleprogramsneedstobefactoredintoplanning.t(#(# 0   7.11Providemoreinformationonthescopeofdocumentationrequirements. L (#(# 0  AswithDMIS,TRWhadsimilarconcernsabouttheamountofeffortittooktoproduce  p managementdocumentation.Theyprovidedthefollowingperspectiveonthisissue: \(#(# 0  0(#(#Inretrospect,whentheentiredocumentationeffortisconsidered,itneedstobe  4  recognizedthattheimportanceofthedocumentationwasnotemphasizedinthe   originalRFPandthereforeassumedtobelesssignificantbyTRWinourproposal.   Wearesureinhindsightthatnoonemeanttominimizetheeffort,butitshouldbe   recognizedthatcomprehensivetechnicalandadministrativeprocedureswere   essentialtorunningconsistentoperationsatthedatacapturecenters.The   operationalproceduresandassemblylinelikeprocessnecessarytohandlethe l formsrequiredallpartiestoknowandunderstandwhatprocedurestofollowand X understandwhatchangesweremadetotheproceduresduringoperationstoensure D optimumproductivity.ItwasalsocriticalthattheDCCadministrativeandfacilities 0 staffshaveguidelinesandprocedurestofollowtoreduceemployeerelationsissues. l And,certaindocumentswereaddedtotheeffortandwerecriticaltotheprogram, X butwerenotinSectionF[ofthecontract].ThesedocumentsincludedDCSC D ManagementInformationSystem(DMIS)documentationandoperations 0 documentationsuchastheOCCCONOPS[ConceptofOperations]andotherOCC  relateddocuments.(#(# 0  TRWrecommendedthatdocumentationrequirementsbegivenmoreemphasisinfuture  RFPs.|(#(# 0   7.12MinimizetheuseofKFP. T (#(# 0  TheAssessmentReportonDataCaptureofPaperQuestionnairescitesinefficiencies ("x  associatedwithKFPduetoitscumbersomesoftwareand100percentvalidation #d! requirement.WithmuchoftheneedforKFPbeinggeneratedbydamagedforms,light $P" pencilmarks,andincompleteerasures,itmaybepossibletodevisemethodsto $< # significantlyreducethenumberofformssubmittedforKFP.%(!$(#(#  0  7.13Producerealtimecostdata. '#&(#(# 0  TheAssessmentReportonDataCaptureofPaperQuestionnairesaddressedtheneedfor )$( currentcostinformation.ThecombinedcostsforDCS2000andDCSCwere p*%) approximately$520million,andcontractmodificationsaccountedforsignificant \+&* increasesovertheoriginalcontractbaselines.Inspiteofthemagnitudeofthedata  capturesystem,realtimelaborexpenseswerenotavailable,norwerecostdataby  operationforeachDCC.Tofacilitatemanagementofthe2010Census,thereport  recommendsusingmoredetailedandcurrentcostdata.t(#(#   XXo8.0  AuthorsRecommendations 8 (#(# Xo X 0  8.10(#(#Implementamorestructuredapproachtodefiningrequirementsforthedata 8  capturesystem.   p (#(#  0  Moreattentionneedstobepaidtodefiningasolidsetofrequirements,especiallybecause H  thedatacapturesystemin2010maywellincorporateadditionalfunctionalityandhavea 4  morecomplexarchitecture.Forinstance,theLockheedMartinPhaseIILessonsLearned   documentspeculatedthat: (#(# 0  0(#(#Additionssuchasdataentryfromwirelessandinternetbasedapplicationsor  complexaddressrecognitioncanhavesignificantpositiveeffectsonvariousaspects  ofthesystemanditsoperation.Itmayalsobebeneficialtolookoutsidethecurrent l boundsofthesystemforopportunitiestoimprovetheoverallCensuscapture X operation.Suchareasofinterestincludetheintegrationofpostprocessingcoding D systemsorusesfortwodimensionalbarcodes.So,inordertotakefulladvantage 0 ofalloftheexperiencesofthe2000Census,enhancementsofthesetypesshouldbe l investigatedandpossiblyappliedtofuturesystemswithsimilarrequirements.X(#(# 0  Thiswillrequireaproactive,disciplined,andsystematicapproachtorequirements 0 definition.Amajormanagementcommitmentwillbenecessarytomakethisareality.  Withoutit,theCensusBureauwillfaceahighdegreeofriskinimplementingdata  capturesystemsandoperationsforthe2010Census.(#(# 0  8.2Researchthepotentialforexpandingthealldigitaldatacaptureenvironment. | (#(# 0  Incitinglessonslearned,oneoftherecommendationsputforwardbyRITRCsuggested P"  that Duetotheuniquenatureandscopeofthecensus,itrequiresanearlyinvestmentin <#! researchanddevelopment.TitanendorsesRITRCsadvice,butsuggeststhatresearch ($x" anddevelopmenteffortsfocusbeyondpaperbasedsystems.Tobesure,thecapabilities %d # andsophisticationofsuchsystemshavesteadilyimproved,asevidencedbytheDCS &P!$ 2000performanceandqualityissuesdiscussedinthisreport.However,thesesystems &<"% stillincursubstantiallaborcostsforprintingandprocessingformsandfornonresponse '(#& followup.Thesystemalsorequiresmajorcapitalinvestmentinscanningtechnologiesin ($' ordertoprocessmillionsofforms.Inshort,paperbasedsystemsarestillcumbersome )%( becauseoftheconversionfrompaperintoimages,andsubsequentstorageinelectronic *%) format.Thisisaverycomplexprocess.+&*(#(# Ѐ0  AlthoughthedatacapturesystemwasunquestionablysuccessfulduringCensus2000,we  havetransitionedintoanerawheninformationispredominatelycollected,stored,  transformed,andtransmittedinadigitalformat.Intheory,futurecensusactivitiescould t beperformedinanalldigitalenvironment.Infact,theCensusBureauisalreadymoving ` towardexpandingtheuseofdigitalmediahavingmadeacommitmenttousemobile L  computingdevicesindatacollectionoperationsinvolvingenumerators.And,the 8  feasibilityofaWebbasedcensusdatacollectionsystemwasprovenduringCensus2000 $ t andwillprovideyetanothermeansofcollectingdatadigitallyinthenextdecennial  ` census.ThisisespeciallytrueiftheCensusproactivelymarketsanonlineoptionforthe  L  2010Census.Numerouselectronicdatafilesbeingmaintainedbystateandfederal  8  agenciesmayprovidemoresourcesofpersonaldatathatcanbeconvenientlyextracted $  fornonrespondents. (#(# 0  AnambitiousgoalfortheCensusBureauwouldbetoeliminatetheneedforprocessing   massivequantitiesofpaperformsbyplanningforanalldigitaldatacaptureenvironment.#XX XXom#Xo XXX   Advancedtechnologiesonthehorizonareintroducingnewformsofdigitalmedia,some p ofwhichareintheearlystagesofdevelopment.Theseeffortsmayidentifyenabling \ technologiesthatcouldfurtherstreamlinecensusdatacaptureoperations.Thismaynot H beachievableintimeforthe2010Census,butresearchcanhelppavethewayforthe 4 transitionintothedigitalworld. p(#(# #XX XXo# XX 9.0  Conclusion XX 4(#(# TheCensus2000datacapturesystemwasaverycomplex,wellmanagedsystemthatemployed 4 advancedtechnologiestoautomatedatacaptureprocesses.Itcapturedmassiveamountsofdata   withinarelativelyshortperiodoftimewithhighaccuracyrates.Howevereffectivethe   automation,itstillreliedonsomemanualprocessingtohandlemillionsofwriteinfieldsthat  posedaproblemfortheautomatedcharacterandmarkrecognitionsubsystems.Thedatacapture  evaluationraisesaninterestingissue:isitworthattemptingtomakefurtherrefinementstothe   automatedcaptureprocessesinordertocapturetheproblematicresponses?Thisissomething l! thattheCensusBureaumaywanttostudyinordertodeterminewhethertheextracostandeffort X"  couldbejustified.Alternatively,improvedformsdesignmayallowmoredatatobecaptured D#! withfewererrors.Manyoftheproblematictypesofresponseswereidentifiedinthe 0$" documentationreviewedforthisreportandarewellunderstoodbyCensusBureaustaff.Perhaps %l # someofthedesignimprovementslistedinthisreportcouldhelptominimizethesetypesof &X!$ errorsfromoccurringinthefuture. &D"% Agoalofthisreportwastolookforcommonthemesinthedocumentationaswellasany ($' conflictingperceptionsaboutthedatacapturesystem.Nosignificantdifferencesofopinionwere )%( detectedduringthereview.Tothecontrary,thereweremanysimilar,complimentaryviewsand *%) findingsthatreinforcedourconfidenceinstatingtheissuespresentedinthisreport. +&* Ї   XX References   XX   Brinson,A.andFowler,C.(2003) AssessmentReportforDataCaptureofPaper  Questionnaires,February19,2003.  Conklin,J.(2003) EvaluationoftheQualityoftheDataCaptureSystemandtheImpactofthe `  DataCaptureModeontheDataQuality,K.1.b,FinalReport,March12,2003. L  Brinson,A.andFowler,C.(2001) ProgramMasterPlan,DataCaptureSystemsand $ t  Operations,March30,2001. `  FormsDesignandPrintingOffice.(2002) FormsDesignandPrintingLessonsLearned,May 8  30,2002. $  LockheedMartin.(2001a) PhaseIILessonsLearned,February21,2001.(IncludesAppendix   Alistedbelowasaseparatewhitepaper.)  LockheedMartin.(2001b) AppendixA,TechnicalLessonsLearned,WhitePaper,Release1 p Version1,February23,2001. \ RochesterInstituteofTechnology.(2002) DCS200DataQuality,v.2.1Final,September20, 4 2002.  p TitanSystemsCorporation/SystemResourceDivision,R.3.d.(2002) Census2000Data H CaptureSystemRequirementsStudy,FinalReport,August23,2002. 4 TitanSystemsCorporation/CivilGovernmentServicesGroup.(2003) Census2000   QuestionnaireDesignStudy,FinalReport,March19,2003(notyetreleased).  TRW.(2001) DataCaptureServicesContractFinalReport,February9,2001.   Weinberg,D.(2000)MemorandumfromDanielH.Weinberg,Subject:ActionstoCorrectPass X"  2KeyingErrorsinCensusSampleMonetaryFields,December7,2000 D#!           XX +&*