Subject: the ldc united nations corpus

in linguist 6 . 973 and a parallel query to the corpora list , rapahel salkie asked some questions about the un parallel corpus from the linguistic data consortium ( ldc ) . before these queries appeared , a response to an earlier query from dr . salkie had already gone out privately from rebecca finch . i certainly urge anyone with experience in using the un corpus to respond to dr . salkie as well . we have also placed a sample of 24 ( out of 21 , 000 ) parallel documents , each in english , french and spanish , in : ftp : / / ftp . cis . upenn . edu / pub / ldc / data _ samples / un _ par _ sample . tar . z these samples should also be accessible , along with quite a bit of other ldc information , from the www page at url http : / / www . cis . upenn . edu / ~ ldc let me add a few words about ldc prices and costs , since dr . salkie 's message expressed the normal human annoyance at being asked to part with money , both in the case of the un corpus and another ( not yet published ) ldc parallel text corpus , the canadian hansard . the ldc membership fee for a university is $ 2 , 000 , and for this fee everyone at that university can get an unlimited and perpetual research license for everything that the ldc publishes during the year of membership . thus you can join the other ninety current members of the corsortium and get not only the forthcoming hansard corpus , but also the other twenty or so databases published this year . for the same amount , a university can get the un corpus and the other 15 databases published in 1994 . whether a particular database , or collection of databases , is worth that amount of money is of course a matter of individual or institutional judgement . we feel that $ 2 , 000 , which is roughly the cost of a moderately configured pc or an international conference trip , is not out of line even for university researchers . speaking for myself , i have a great deal of sympathy for the effort to provide research resources free or at minimal cost , and i have been involved in several successful efforts to bring out such databases over the years , including the acl / dci cd-rom , the eci disk , the celex disk , and others offered in the range of $ 25 - $ 200 . these efforts rely heavily on volunteer labor and other donated resources ; in several cases they have also relied on cash donations from the ldc . however , volunteer labor is rarely available in the needed quantities ; and of course ldc - supplied cash , as well as the existence of the ldc as an organization , depends on income from somewhere . the money that we get from memberships and database sales is a crucial part of the picture - - - without it the ldc would not exist , and neither would either of the databases under discussion . to highlight the point , the history of the u . n . publication is worth reviewing briefly . we decided to try to publish the u . n . archives because translation researchers wanted parallel texts . after concluding several months of negotiations with un representatives and lawyers for both sides , we paid for a nj - based computer consultant to go into the un offices at night so as to make backups of the archives from dismountable disk packs for a long-obsolete wang word processor onto cartridge tapes . this required several months and cost a considerable sum ; we had to use this particular person because he was an authorized service rep for the un facility . then came six person-months of work at the ldc . we had to decode the proprietary and undocumented wang backup format , and the equally proprietary and undocumented wang character set , typographical codes and file structures . we re-organized the entire archive and translated it into wordperfect format , and published a certain number of cd-rom s in this form for the purposes of the un - - - this was part of the agreement that we made with them for access to the data . then we translated the documents into iso-8859 - 1 with sgml markup ( including a working dtd , for those how care ) , and worked out the correspondences among documents . this was far from trivial , since each un language had been entered separately , with no coordination of file names , file dates , or even division of documents into files , and there were tens of thousands of documents per language . this work was mainly done by dave graff , whose salary the ldc pays . we are not likely to recover the costs of acquisition and production of this database through sales and memberships bought for its sake . we subsidize our members by cost-sharing with government grants , or by using income from more popular or less expensive databases to cover unrecovered costs of less popular or more expensive ones . in the case of the forthcoming hansard corpus , which dr . salkie also mentioned , the cost of acquisition and publication has been similar to that of the u . n . material , and the same remarks about subsidies apply . whether a particular database is worth a certain price is a matter of individual taste , but as a matter of simple arithmetic , the fees charged in these two cases are unlikely ever to cover the costs incurred . for those who have read this far , i would like to repeat a standing offer that has been in existence since the beginning of the ldc . if you are interested in cd-rom publication of a language-related database that is plausibly of interest to our membership , and this database is reasonably close to being in shape to publish , we will pay the costs of production , using your label design or one worked out with you ; we will give you up to a hundred copies to do with as you see fit ; we will put the item in our catalogue at whatever price you choose ; and we will remit to you any resulting income in excess of our production costs . the copyright ( if any ) will remain with you , and we will handle any user license arrangements that may be necessary , sending the signed licenses to you . we have published several databases on this basis , and are planning to publish several others , although ( from past experience ) the chances of making back our production costs are no better than even . best wishes , mark liberman myl @ unagi . cis . upenn . edu 619 williams hall university of pennsylvania phone : 215-898 - 0141 philadelphia , pa 19104-6305 fax : 215-573 - 2175
