Merit Rewiew, Digital Library design and cooperative cognition [1].
W.A. TURNER, P. de GUCHTENEIRE, K. van METER
W.A. Turner : Infometrics Research Center, CERESI-CNRS, 1 place Aristide Briand, 92195 Meudon.
P. de Guchteneire : UNESCO-MOST, 1 rue Miollis, 75015 Paris
K. van Meter : LASMAS-CNRS, 59 rue Pouchet, 75017 Paris, France
Paris, 1995
 |
A better understanding of the general economics of information flows over
Internet is needed in order to develop strategies for digital library design. Internet is not just a new information marketplace in need of rationalization in order to overcome the current dysfunctions of early infancy. It is more importantly to our mind an arena for building science with digital libraries serving as external memories to support the social processes involved. Our argument will consequently be the following : at the present time, the dominant model for digital library design is the market model but, in fact, information flow economics over Internet depend much more upon the social and organizational dynamics of cooperative cognition in collaboratories [2].. We will show that the meaning and importance of"peer review" stems from the market model and that "merit review", as an alternative concept, is useful for designing digital libraries to support work of cooperative cognition. |
Introduction" The electronic wave is about to hit us ", this in a nutshell is a summary of a recent article that appeared in Science on electronic publication (Taubes, 1996). Over 100 peer-reviewed journals in science, technology and medecine were on Internet at the end of 1995 and the Commission of the European Community estimates that by the year 2000 about 1/3 of revenues in these different sectors will derive from electronic publishing (CEE, 1993). The wave has been building up for many reasons among which the most often cited are the increasingly prohibitive cost of paper journal subscriptions ; the desire to cut down publication delays ; and the growing dysfunctions for authors of the imperfect, oligopolistic marketplace for scientific papers which is now dominated by a small group of publishers (Odlyzko, 1994 ; Harnad, ftp.). If there is still a lot of discussion about what the impact of the wave will be on the underlying processes of scientific production, one point seems increasingly taken for granted : peer-review is a useful technique to help manage the information flows of the electronic era.
This paper will look critically at this issue using debates in the sociology of science as background for discussing our work underway with UNESCO aimed at building a digital library of documents relating to the Management Of Social Transformations (Unesco's MOST programme). These debates suggest the need to substitute the wider notion of " merit review " to the more restricted notion of " peer review " in contexts such as MOST, where the central concern is linking Science, Technology and Society. The point is a general one : at the present time, we seem to be locked into a heaven or hell type of alternative with peer review as the solution for conciliating the two extremes. Computer-mediated communications will be heaven if cheap, easy access to up-to-date material is a guaranteed feature of new inforoute infrastructures ; they will be hell if too much effort has to go into filtering out the infojunk circulated over the system. More attention could be paid to how digital libraries are currently being designed in a wide variety of institutional settings to promote context-specific knowledge production practices. The UNESCO programme will provide us with one example of different, perhaps more interesting questions that have to be raised.
1.
From Peer Review to Merit Review
The link between knowledge and action in a research field is generally established through peer review. The peer review process is a way of collectively organizing the distribution of resources amongst competing knowledge production strategies. Scientists evaluate the technical skills of their colleagues using basically the same theoretical, methodological and data interpretation frameworks to decide upon the interest of individual research strategies. The rules of the game are generally clear for these on-going micro-evaluations of skills and strategies. In MOST, however, the rules of the game are not clear and have to be defined. Social transformation research doesn't yet exist as a specialty: for the moment, there is not a normative socio-cognitive framework to organize collective behavior and provide quidelines for producing and evaluating new science. The border-line separating "in-network" from "out-network" research is not clear and this makes the goal of correctly delimiting a problem space for scientific investigation and action difficult to achieve.
At a more macro-level, this general problem of delimiting the field of social transformation research is complicated by a political issue. If we look more specifically at the cities research network of the MOST programme, participating laboratories are located throughout the world. For example, the Clearinghouse prototypes under development at the present time will be tested in six sites located in Egypt, South Africa, Brazil, Canada, Spain and France. There is no reason to expect that urban research in Cairo will take up the same questions, use the same methodologies and interpretation frameworks as urban research in Buenos Aires, Paris, Capetown, Granada or Vancouver. On the contrary, it is probably important to respect the differences. But for the MOST programme, this raises the question of resource allocation policy. Two alternative options can be envisaged: either the vested interests of stakeholders throughout the world are recognized, in which case the policy adopted will likely become one of "dividing up the cake", that is, one of allocating the available resources on the basis of some regionally-based, pro-rata formula designed to respect the differences between North and South, East and West; or, technical evaluation criteria will be used to class research proposals as mainstream, secondary or of marginal interest. Neither alternative is well-suited to the MOST context. The first leaves little hope that research will be considered sufficiently independent to provide a useful perspective on the relationships between knowledge and appropriate political actions. The second aims at establishing the research community's autonomy but encounters the limits of the peer review process outlined above : the rules of the game are not clear; a consensus about what is important and what isn't still has to be built in the field of social transformation research.
We suggest using the notion of "merit review" as a substitute to "peer review" in the MOST context. Merit review is a notion that applies to situations where game rules have to be clarified, where the problem is not one of applying a normative framework to evaluate works, but rather one of managing the social dynamics leading to the definition and adoption of a normative framework (NSF, 1986). Merit review focuses attention on the difficult task of drawing, managing and maintaining over time the border-line between "in-network" and "out-network" research. This task precedes peer review in the sense that it is only by supposing that a set of limiting conditions is given that the technical capacity of research teams to contribute policy relevant social science research into the dynamics of social transformations can be evaluated. If peer review is defined as a technical evaluation of research capacity, then merit review is defined as the process through which consensus is built up and maintained over time on the appropriate framework for carrying out this technical evaluation. It integrates the political dimension of the evaluation process in that it aims at organizing discussion around differences, and seeks to provide benchmarks for evaluating the specific interest of these differences given the self-organization dynamics of the research network.
We now have questions for empirical research on collaboratories which concern the dynamics of fence-building and fence-tending over Internet. What communication patterns characterize efforts to build and maintain the borderline between " in-network " and " out-network " research? Three levels of analysis can be identified : knowledge claim production in " in-networks " ; empowerment of new " out-network " stakeholders to participate in this local, knowledge claim production process ; and, finally, the nature of social interactions at the borderline. Our research programme consists in, first, trying to better understand the dynamics involved at each of these three levels ; second, in asking how merit review, as a cooperative cognition practice, structures these dynamics ; and, third, in building a suitable economic model for information flow management in Internet based on our understanding of the social and organizational dimensions of cooperative cognition. This paper will only look at the first two points.
2.
The in-network research context
Traditional sociological theory holds that the laboratory plays a gatekeeping function in the social system of science by deciding what must be kept secret and when results in the making can be published. Power of decision is in principle concentrated in the hands of the Laboratory Director who knows just how dangerous for reputations, patent approuval and competitive advantages the unwarrented disclosure of knowledge claims can be. Internet, it is said, has weakened the power structure of laboratories in the sense that central control over the in-house review process leading to disclosure is increasingly difficult to enforce. Email, Newsgroups and World Wide Web sites are used by individual members of a lab, and they have complete discretionary power on what should be disclosed and what should not. In fact, it is probably exaggerated to speak of open disclosure and free communication practices : the craft culture of science developing over Internet is probably better described using a distributed authority model instead of the more traditional gatekeeping model presented here. We will first show how Internet technology can be used to encourage distributed authority both at the group and organizational levels of collaboratory life and then go on to use the model to design a digital library acquisition strategy.
Informal Email exchanges characterizes the most active aspects of intellectual life in collaboratories. Email services of Internet (file transfers, mailboxes, etc...) offer specialized groups the possibility of interacting at a distance in order to compare notes, share texts and build interpretive frameworks for their data. These groups are generally formed by people who know and have confidence in one another ; they are built on empathy and remain largely closed to the exterior to the extent that cries of alarm are being heard, not about information overload, but about technology-supported specialization and the threat to pluridiciplinary research (Economist, 1994).
Empathy groups are stakeholders in a particular institutional setting, defined not only by the cognitive content of the research carried out, but by a variety of organizational arrangements linking that research to industry and society at large. Efforts to understand these arrangements often focus on the problem of expertise given the need to organize fruitful interactions between the content-specific skills of scientists and the broader, social-specific skills of their partners. We've decided against this a priori assumption of skill types, preferring instead to take seriously the idea of computer-mediated collaborations in order to raise the following empirical question: can people with different backgrounds develop concrete arrangements for working together over Internet? The variety of technological options available for managing mailing list distribution of files and messages are currently being studied in order to answer this question in the MOST context :
3.
The out-network of a research programme
In-networks are composed of empathy groups and stakeholders. Empathy groups in terms of the MOST context are those groups working in the same geographical context on cities in Egypt, South Africa, France, Canada and South America ; they are also stakeholders in the MOST programme in that they share the general concern for building a suitable analytical framework for understanding the dynamics of social transformations in urban centers. Integrating the results of individual case studies into a global analytical framework requires socially recognized cognitive norms which do not exist for the moment, but which could use Internet exchanges as a means of defining them. We tried to show above that Internet technologies offer appropriate tools for organizing stakeholder interactions to fulfill this capacity building function. But now it is important to look critically at a model which might have been suggested by our distinction between in- and out-network research.
The model seeks to describe scholarly communication as a link between the context of discovery - the tinkering, manipulating and hands-on aspect of laboratory life- and the context of validation in which knowledge claims are submitted to peer review before general dissemination as established facts into the public domain (Holton 1973). We do not accept the linear reasoning this model implies : knowledge claims are not built in a given context, evaluated in a second and then disseminated in a third. Stakeholders are in all three contexts at once, the only distinction our model admits is between those who actively take part in a programme and those who have not (yet) been admitted to participate.
Leveling criticism at the linear model has important consequences for the design of a digital library because traditional thinking in the library sector generally takes for granted the distinction between production, evaluation and use of knowledge claims. Library services are normally designed to offer access to a pool of knowledge : if discovery takes place in a complex, contingent, and emotion-laden social landscape, peer review extracts knowledge claims out of this messy, human environment to make them "context free". This gives value to document stores because the information made available through them can theoretically be used for general application in all situations : verisimilitude, it is argued, has nothing to do with conditions of information production or use, and through peer review the verisimilitude of knowledge claims made in publications is established. Libraries store documents that enter the public domain after peer review, and can be used as a resource in any number of new situations; but if this model is criticized, what document acquisition strategy should be used ?
Our answer to this question lies in the application of the distributed authority model introduced above: document storage is considered to be a local responsability of individual stakeholders participating in the system. But who are the stakeholders ; how does their membership evolve over time ; and what consequences will this evolution have on information available through the distributed storage system of the digital library?
The services of the MOST Clearinghouse include : a Digital Library Database that contains the research results relative to social transformation research around the world ; Mailing Lists that are designed for stakeholder interaction through MOST ; an Events Agenda to inform Internet users about UNESCO initiatives in the area. This Events Agenda is now operational and can be accessed over Internet at the following address: http://www.unesco.org/most. Four functions- document production, consultation, annotation and suppression- can be applied on each of these three databases, but different levels of empowerment have been implemented in order to ensure design coherence. Not everybody in the system has the same rights.
For example, visitors who hit upon the " Events Agenda " while browsing the information stores of Internet will discover, among other things, calls for research proposals, summaries of past events and announcements of meetings and conferences to come. If the system is unable to identify visitors in the MOST Register, they will be invited to sign up as potential partners of the MOST enterprise. A special form is automatically sent to them over Internet inviting them either to add their Email address to the list of subscribers to the "Events Agenda" or define their centers of interest for inclusion as subscribers on the MOST mailing lists. Subscription means in both cases that of the four functions mentioned above, only the consultation function is open to them ; if they want to produce or annotate existing documents, they have to make a formal request to the MOST secretariat for stakeholder status. Finally, the suppress function is only attributed to the appointed administrator of an empathy group. In the decentralized architecture of the MOST Clearinghouse, nodes are being set up in a variety of geographical locations in order to capture the local, situated nature of urban research and to manage the language problems inherent in working world-wide (Spanish, French and English are the working languages).
We talked above of the North-South, East-West decision matrix of UNESCO and the pressure generated by this matrix to push resource allocation decision-making in one of two directions, neither of which is satisfactory: "context without content" where political considerations dominate the allocation process; "content without context" where it is subordinated to "context-free", technical criteria of scientific verisimilitude. Reflexivity implies a << content in context >> perspective, the idea being that efforts to evaluate content serve to fix the limits of a working environment while, at the same time, the definition of these limits determine the meaning of content. Digital library design based on merit review implies putting knowledge into action.
Libraries are generally seen as << pools of knowledge >> whose task it is to ensure access, information transmission and use in all types of situations. Distributed digital library design implies taking a different position : we've argued that knowledge is not << context-free >> but produced through language games which are grounded in local, situated actions. Stakeholders who are admitted to participate in the decision-making process are in fact spokespersons for local opinions and, consequently, it is legitimate to ask, before evaluating their knowledge claims, if their positions are representative of local realities. A first requirement for digital library design is to be able to answer this question of political representativity. A second requirement concerns the content of documents circulated over the distributed system. It is not enough to establish their political representativity ; the goal is to evaluate their impact on existing resource allocation strategies : should new cognitive orientations be financially supported to develop social transformation research or not ? A third requirement is to recognize that in the cooperative cognition framework we are defending, knowledge has to be understood as a tool for calibrating future actions, and that the condition for achieving this understanding is to organize language games.
4.
Merit Review : putting knowledge into action in
Our 1996 work programme consists in building a merit review process into the digital library design of the MOST Clearinghouse. It is consequently too early to provide concrete results and all we can do here is to indicate the procedures we will use in our work. These procedures are based on the assumptions outlined above :
Our partners in the MOST digital library design initiative are, as we said before, empathy group administrators working in different geographical locations around the world. For the moment, they are the only stakeholders recognized by the system in their different regions, but potential partners have also been identified. This creates the experimental conditions needed for focusing the merit review process on integrating new partners into the << in-network >> of MOST as stakeholders in the decision-making process. We have built a situated action context in which to organize language games and study the cooperative cognition mechanisms used for fence-tending and fence-building in collaboratories.
We defined the craft practices of scientific endeavor in reflexive terms : social acceptability of knowledge claims is decided through tinkering, adjusting and testing verisimilitude in an environment dominated by certain socio-cognitive norms ; but each document is, in fact, an interpretation of what knowledge claims can be successfully advanced and defended in that environment. Scientific writing can be represented as a problem of avoiding two pitfalls : that of advancing claims that are completely disconnected from on-going practices ; that of simply reproducing what has already been done. Our mathematical models serve to measure degrees of ressemblence between an existing state of knowlege claims and new ones made in newly circulated documents (van Meter et al., 1994 ; Turner et al., 1995 ). << Existing state >> is defined by the information available in the document store of an empathy group administrator (that is at a particular machine node, geographically located, in the distributed storage space of the MOST digital library). The information is represented by a set of simple statistics : N is the number of documents in the existing state at t ; Ci is the frequency with which a given subject is mentioned in N ; Cij is the number of times that subject is cited in association with another subject. Changes in existing state are measured at t+1 by using t as a baseline ; they concern differences in N, Ci and Cij over time and are interpreted as indicating an evolution in centers of interest. This evolution is attributed to stakeholders when << in-network >> publications are analyzed ; or to potential partners when << out-network >> publications are analyzed.
In MOST, when potential partners are identified- or have themselves subscribed- for membership as stakeholders in the social transformation collaboratory, they are asked to supply the Clearinghouse with five of their most recent publications. These publications are not submitted to peer review, but are analysed in the way we just described. The goal is to determine their impact by measuring the extent to which the subjects discussed (Ci) and the associations introduced (Cij) converge with << existing state >> stakeholder information. Different cognitive strategies identified in this way enable the MOST secretariat to raise the political issue of stakeholder representativity as spokespersons for a regional urban research context. In other words, conflicting cognitive strategies serve to launch the merit review process aimed at deciding upon criteria of admission of potential partners to the << in-network >> of collaboratory stakeholders.
It is perhaps important to insist once again on the fundamental difference between peer review and merit review as presented here : merit review uses the notion of conflicting cognitive strategies as an invitation to modify existing game rules in order to enable new players to participate in the game ; peer review, on the contrary, will tend to exclude players who do not play by the rules.
We explained above that local viewpoints are needed to identify significant events that merit attention in the global context of MOST : different cognitive strategies in play locally (<< events >>) generate a list of subjects (Ci) and a limited set of association patterns in which these subjects are embedded (Cij) ; these local, event-generated lists of subjects and association patterns define the boundary objects of MOST. We take for granted that debates about appropriate courses of action for urban research in Cairo will focus on questions embedded in different subject structures to those encountered in Buenos Aires or Paris. In other words, boundary objects focus a great deal of ambiguity, but at the same time, they can be understood as objects of interest to social transformation research in general. Indeed, one noted feature of boundary objects is that they are simultaneously local and global, common and specialized, shared and segregated (Star and King, forthcoming) which means that despite their ambiguity, arrangements can be worked out which will allow people to organize cooperative ventures around them. The MOST infrastructure needs mechanisms for working out ambiguity.
Conclusion
Our main criticism of the proposal is that its force derives from a false perception of the knowledge production process. When the question focuses on how "context-free", objective knowledge claims are extracted out of the messy, contingent, emotion-laden environment of idea discovery and problem formulation, a process of socialization is generally postulated and peers are assigned the gate-keeping role of enforcing socially-accepted cognitive rules and routines. However, this socialization hypothesis ignores the issue which is increasingly coming to dominate research in the field of cooperative cognition (Bowker et al (ed.), 1996): how are sets of cognitive rules and routines made socially acceptable in the first place? Increasingly the answer is seen as lying not so much in the concept of rule enforcement as in the idea of open-ended games : collective action implies that the rules of social acceptibility are modified in the course of playing the game ; efforts are constantly being made to accomodate new cognitive strategies and new players. Merit review was employed in this paper to identify and analyze the dynamics of these open-ended games.
Research is currently underway which is aimed at developing appropriate models for understanding the general economics of information flows over Internet and, clearly, these models will influence thinking about appropriate strategies for digital library design. The dominant model at the present time is a << market model >> and we've tried to show the extent to which peer review and the pool of knowledge design strategy depends upon that model. In proposing an alternative model - the << cooperative cognition >> model - our goal is to focus design on managing the social and organizational dimensions of the merit review process underlying collective knowledge production practices.
Acknowledgements :The authors would like to thank Mr. Y. Muller, researcher with the Centre de Recherches sur l'Habitat at the School for Architecture in Paris, who organized contacts for us with scientists working in Cities Research and, in his capacity as Head of the URBALAB Documentation Network, assured the liaison with the PIR-Ville Programme of the CNRS through the active cooperation of Mr. F. Godard, the Deputy Director of this Programme.
Références :
© "Solaris", nº 3, Juin 1996.