To improve the Web-based architecture of the previous version of DeGeL, we developed a new version (DeGeL-II) of the digital library. When designing the new version, our goal was to create a distributed, Web service- based, open architecture implementation according to the service oriented architecture (SOA) [41] design specification. The SOA architecture, which is also used by other initiatives such as SEBASTIAN [32], SAGE [29] SANDS [42] and CDS consortium [43], has the power to distinguish between different parts of the system such as the knowledge repository, the knowledge specification tools and the run-time applications engines.

This new design provides the ability to develop a suite of tools for guideline specification, retrieval and application. These tools use the services of the central digital repository but are not dependent on it. The open architecture may also host alternative tools for guideline specification and application. DeGeL-II's server allows development of rich client tools by using Web-service methods to retrieve and edit guidelines in the knowledge-base.

The internal architecture of DeGeL-II server was assembled from the following five modules: (1) a guideline database that contains the overall schema to support the hybrid multiple ontology representation; (2) a module responsible for content management; (3) a new guideline search engine that replaces the previous search engine [39] and supports full-text, context-sensitive and concept-based searches for enhanced guideline retrieval; (4) an authorization and authentication module that supports the group-based authorization model; and (5) a Web-service API that enables the guideline knowledge-base server to accept client requests and to orchestrates multiple steps in order to perform the requested transactions.

As DeGeL supports storing and retrieving guidelines from multiple ontologies, it uses several standards to represent the specification languages. All entities within DeGeL, such as documentation ontologies, specification ontologies and guidelines are stored within a central database using a relational schema. The schema allows the library administrators to define a hierarchical structure for each of the ontologies. This hierarchical structure is used to represent the ontology's hierarchy of knowledge-roles. For example, the Asbru ontology includes hierarchy nodes for the guideline's intentions, plan-body, effect and conditions. The conditions node includes sub-nodes for filter, setup, complete, abort and other conditions. When a new guideline is created the schema stores data to describe each of its plans, which can have details according to all knowledge roles at multiple representation levels. The internal structure of the data that is stored in each knowledge role is defined by the Semi-Formal and Formal schemas of each specification language. Each specification language is defined by an XML schema that is used by the knowledge specification tool to validate the content of the knowledge roles.

In addition to the library server, we have developed several external tools to administer and maintain the guideline library. These tools include: (1) An ontology-builder tool for the specifying and maintaining the hybrid ontologies stored within the knowledge-base server. (2) An authorization-specification tool which was developed to allow the library administrators to create and manage groups of users with different profiles consisting of a set of library roles. (3) An axes-builder tool which was developed to create and maintain the semantic axes of medical concepts used to classify (index) guidelines. These semantic axes are needed to support the enhanced concept-based retrieval mechanism of the guideline search engine.

The two main applications that are part of the overall architecture and use the library server are the guideline runtime application engine [40], which is outside of the scope of the current paper and the guideline specification platform, Gesher, which was developed in the current study and is described in detail in the following sections.

To support the task of structuring the consensus, Gesher provides a graphical interface (see Fig. 2) that is used by an expert physician, assisted by a knowledge engineer, to roughly structure the main procedural aspects of the guideline consensus. When using the tool, the users can choose plans (i.e., Asbru's notification for a clinical step) from different semantic types and add them to the guideline's procedural flow. These clinical semantic types of plans include procedures, drugs, observations, educational steps, follow-ups, decisions (e.g., if-then-else plan in Asbru), general plans and reference-plans (i.e., reference to existing plans). In addition, the expert can also mark a clinical step as a periodic step, meaning that the step should be performed more than once.

Fig. (2). The hierarchical plan builder in Gesher that the expert physicians use for specifying the procedural aspects of the guideline. In this case the "Management and Treatment" plan, of the PET guideline, composed of evaluating the patient's state (severe or mild preeclampsia) and applying the matching treatment. The user selects plans from different types (frame 1) and adds them to the hierarchical flow chart (frame2). For each plan, several properties can be defined (frame 3); for composite plans the procedural aspect are specified (frame 4). The sub-plan hierarchy is displayed in a tree-view display (frame 5). In this phase the expert also defines a list of declarative concepts that will be further specified in the following phases (frame 6).

Since most of the approaches for guideline specification contain a hierarchical structure to represent the overall complex guideline, Gesher provides the user with the ability to transform each plan into a complex plan consisting of a set of sub-plans. A new diagram will then be created for specifying the new complex plan. When sub-plans are being added to the diagram, the hierarchical structure of the guideline is being constructed. Gesher provides the ability to explore this hierarchy of plans and sub-plans in several visual ways such as by navigating through the tree-structure. This makes it possible to search for plans by their titles, or by navigating through the control flow diagrams. The hierarchical structure of the guideline is immediately stored in DeGeL (using the Web-service based API).

The consensus includes specifying the declarative aspects regarding the clinical algorithm. These aspects include the procedural type of a complex plan (i.e., one that has more than one sub-plan) including: "Parallel" execution of sub-plans; "Sequential"; "Any-Order" (i.e., plans are executed in parallel but no order defined); "Unordered" where the execution of sub-plans may overlap; and "Or" where not all sub-plans are mandatory. The declarative aspects also include specifying the eligibility criteria for starting each sub-plan; the abort criteria to cancel the plan's execution; the complete criteria for successfully finishing the plan; a notation whether the success of completing the sub-plan is mandatory for the success of completing its parent plan; and for each of the complex plans, a notifications of the number of the sub-plan that is mandatory for completing successfully the parent plan. In order to assist the user in structuring the consensus, default values were selected for some of these declarative properties. For example, the default value for the procedural type of complex plans is sequential while sub-plans are mandatory by default (except when the parent's procedural type is "OR").

Another task performed in this step is to create a list of the declarative medical concepts that are related by all the sub-plans comprising the guideline and to provide a textual description for each of these concepts. This list of medical concepts is called the “Guideline Declarative Knowledge”, and will be further specified and defined, in the next step of the specification methodology.

In this step, a clinical editor uses Gesher to create a complete Structured-Text representation of the guideline, according to the knowledge-roles of the specification ontology. Although we used the Asbru language, this step is not restricted to particular guideline ontology. Using Gesher, the clinical editor refines the plans created in the consensus and links these plans to portions of text in the source guideline. This creates a textual representation that is detailed according to the knowledge-roles of the target ontology and is performed for each of the guideline's sub-plans.

When the clinical editor marks-up text from the original guideline, the reference to the particular text segment in the source guideline is saved in the guideline library. These back-pointers that link the hybrid content to the sources are in the library and used by the interface to mark the location in the source when editing a knowledge role of a specific plan. Marking the text can assist the user when creating the specification or when updating a guideline at a later point in time. To edit the content of the structured-text representation (Fig. 3), Gesher provides a rich HTML editor, and help the user to mark-up a text by dragging portions of labeled content from one or more source guidelines into the selected knowledge role frames.

Fig. (3). The interface used to edit the Structured-Text Representation. The clinical editor selects a concept from the "Guideline Declarative Knowledge" (frame1), and then marks text from the source guideline (frame 2) to add it to the text editor in order to create the specification of the selected concept.

In addition to specifying the optional text of each knowledge role of the sub-plans comprising the guideline, the clinical editor performs additional tasks such as classifying the guideline according to the semantic indices of the digital library and connecting raw data concepts and declarative the knowledge concepts, to concepts from available, standard medical vocabularies, such as LOINC, CPT and ICD.

In this step, the expert physician and the knowledge engineer use a graphical interface within Gesher to further specify the consensus for the declarative aspects of the guideline. These aspects, which were textually described in the previous step of the process, are now specified in a detailed manner that we call semi-formal representation. The idea of this representation is to describe the declarative concepts in a complete manner that will later allow the knowledge engineer to create the formal and computer interpretable representation. To implement this action, we developed a specific interface called the Knowledge Map that allows the medical expert, in a reasonable amount of time, to examine and confirm all the concepts described in the guideline knowledge.

The knowledge map tool provides the ability to organize and describe the concepts within a specific guideline, at a level of detail that will later allow creating the formal representation. Each concept in the guideline is described by several attributes that are available in the graphical interface. These attributes include a textual description; a description of the type of concepts; a description of the possible values; and a definition of temporal aspects such as the period during which a certain measurement of the concept value is valid in the context of applying this guideline (i.e., for how long is a specific measurement valid).

To keep the knowledge map easy to understand and usable for expert physicians, we had to consider the trade-off between the expressiveness and simplicity of the underlying semi-formal model. To support cases in which the concepts needed to be described, complex constraints that exceed the expressiveness provided by the semi-formal model (e.g. composite temporal patterns or complex mathematical calculations), the expert can use a textual explanation in which these complex constraints should be described. The textual description allows the knowledge engineer to later specify these constraints in the formal representation.

The semi-formal model includes several types of concepts that can be added for describing the guideline's knowledge. The following section describes these types:

Primitives describe the raw data which is collected and examined during a clinical procedure. The primitive concepts may be numeric, Boolean (true/false) or symbolic (e.g., low/normal/high). The white blood cell count is an example of a primitive parameter. Most of the primitives will also include a time-stamp to specify the date and time of the measurement.

Dates of Events describe raw data that includes a time stamp without any scalar value. "Date of Birth" or "Last Menstrual Period" are examples of this type of concepts.

Abstractions describe composite concepts that relate or derive from one or more other concepts. Most of the concepts within free-text source guidelines are abstract concepts. For example, a concept such as "Thrombocytopenia" is a Boolean concept (i.e., it can be evaluated as true or false for a given patient in a given point of time) that is abstracted from "Platelet count". In the PET guideline, Thrombocytopenia is defined as Platelet count < 100,000 per mm3. When describing an abstract concept, it is necessary to describe the relation(s), which we call "Abstracted-From", that involve other concepts and logical constraints on their value or temporal constraints on their time-stamps. When the abstraction is composed from more than one concept, it should also include a description of the logical operators that should be applied. Another example of an abstract concept from the PET guideline is "Elevated Liver Enzymes", which is defined as SGOT>60 AND SGPT>60. Notice the value constraints (">60") and the "And" logic operator.

Logical Operators and Functions. Another type of knowledge element, available in the knowledge map, is the Logical Operators and Functions. Elements of these types are used to describe the logical operator to apply when a concept is abstracted from more than one concept. The logical operator types include elements to describe the relation of AND (e.g., "all of the following"); the relation of OR (e.g., "one or more existing of the following"); and the relation of K-From-N (e.g. at least two are present from all of the following). The function type is used to describe mathematical functions to compute on the "Abstracted-From" concepts. When adding a function to the knowledge map, it is necessary to express it in the textual description.

To illustrate the knowledge map interface and its underlying semi-formal model, we use the following example which is taken from the PET guideline. Fig. (4) illustrates the graph for specifying the concept “Mild PreEclampsia”, which, according to the guideline’s text, is defined as: “Blood pressure of 140 mm Hg systolic or higher, or 90 mm Hg diastolic or higher, that occurs after 20 weeks of gestation in a woman with previously normal blood pressure and Proteinuria, defined as the urinary excretion of 0.3 g or more, in a 24-hour urine specimen, OR a level of +2 or higher in a protein dipstick”.

Fig. (4). A knowledge map graph specifying the concept “Mild Preeclampsia”. The abstracted-from relations are expressed by the directed links in the diagram. The concepts and their relations are graphically displayed in the map area (frame 1). When selecting a concept, the structures-text description is displayed (frame 2), in this case for the Proteinuria concept. For defining the constraints on the concept’s values, the editor use the interface on the bottom left area (frame 3), in this case “> =2+” and “> =0.3g”.

In this step, the knowledge engineer uses Gesher to create the formal representation in the final executable, target guideline ontology. Because we have chosen in this study to use the Asbru ontology as the underlying formal specification language, we developed a module that automatically generates Asbru guidelines (see Fig. 5) from the inputs of the previous steps of the specification. By using the products of the former steps of the methodology, this module creates the representations of the guideline according to the syntax of the Asbru language and immediately stores them in the library.

Fig. (5). The interface for creating the formal representation. After selecting a plan from the sub-plan hierarchy (frame 1), the user selects a knowledge-role from the ontology (frame 2) to examine the Asbru XML (frame 3) that was automatically generated. The user can also use the source guideline (frame 4) to link each element of the guideline to the source recommendations. The user can also use the run-time application engine to debug the guideline and correct it when necessary.

Fig. (6). The component used to manually perform the guideline semantic classification. The user can search for classification concepts (frame 1); explore them in the axis structure (frame 2); and use them for the guideline classification (frame 3). The panel on the left bottom (frame 4) displays the hierarchical locations of the assigned.

Fig. (7). Interface for interactive exploration of hierarchical diagrams of clinical guidelines. The sub-plan hierarchy is presented in a tree-view display on the left (frame1); the interactive flow-chart diagram is displayed on the right (frame 2).

Fig. (8). Reusing existing specifications. The figure demonstrates the option of copying a sub-plan between two guidelines that are opened simultaneously in Gesher. A composite sub-plan from an existing guideline (frame 1) was copied to a new guideline (frame 2).

Fig. (9). An example of the layout of the Preeclampsia declarative knowledge, organized by an expert physician (frame 1), compared to the layout of the same guideline knowledge, as organized by a knowledge engineer (frame 2).

The task of the knowledge engineer when using this module is to link between the procedural and declarative aspect of the guidelines and then to generate and validate the formal representation, which, in case of Asbru, is represented in XML. To further validate and, if necessary, to correct the guideline, the knowledge engineer can run a simulation of it by using the runtime application engine that is integrated within Gesher in a mode that we call "debug mode". To examine the behavior of the multiple paths within the guideline, the user utilizes the interface of the runtime application engine to provide simulated results regarding the different medical concepts and to observe the corresponding behavior of the system.

Despite the fact that this step is specific to a single ontology and that the module we developed supports only the Asbru ontology, Gesher can be easily extended to support the generation of formal representations according to other target ontologies. For example, a plan that was declared in the consensus as a decision plan is translated to Asbru as an “if-then-else” plan. However, if GLIF is being used, it can be translated into a decision-step plan. Although the design of the internal architecture of the system supports such extensions, nevertheless, an additional development will still be needed.

To summarize the specification process and to illustrate the overall methodology, we describe here the cycles of the guideline specification for diagnosing and managing preeclampsia/eclampsia toxemia (see section 4.1.1). The guideline was selected by the head of the OB/Gyn division at the Soroka Medical Center due to its high occurrence and risk, and the specification process was performed in collaboration with experts from Soroka. To form a clinical consensus two meetings were held with senior medical staff from the hospital, including the participation of three ward managers and two senior experts. The consensus, which included some customizing to the original source, was documented and used in the specification steps.

After a training session, one of the medical experts decomposed a skeleton of the guideline using the Gesher tool in an iterative fashion that included four meetings of a medical expert and a knowledge engineer. Each meeting focused on separate sections of the guideline (diagnosis, mild preeclampsia management, severe preeclampsia management, and eclampsia management). The decomposition of the guideline in Gesher was completed, and then validated in a meeting with senior medical managers.

In the next step, an intern was trained as a clinical editor and used Gesher to create the structured-text representation and to link between elements in the structured guideline and the original source. To complete this task, the clinical editor consulted with the knowledge engineer during the guideline structuring process to further understand Gesher and the semantics of the representation model.

During the next step, the medical expert was trained to use the knowledge-map interface for creating the Semi-Formal representation. To shorten the time for completing this part of the process, the semi-formal representation was actually created with the graphical tool by the knowledge engineer using the structured text representation. However, after only two one-hour sessions the medical expert was able to validate and approve all elements in the specification. Because the specification of the structured-text comprising the recommendations and customizations was very clear, there was no need to involve again the senior ward managers.

The formal level was completed by the knowledge engineer who used Gesher to generate the guideline in its formal Asbru-based XML representation.

To evaluate the Gesher system and the guideline specification methodology, we conducted a study that included specifying the guideline for diagnosing and managing preeclampsia/eclampsia toxemia (PET) [44]. PET is a condition occurring in pregnancy characterized by high blood pressure and the appearance of protein in the urine. The condition is highly dangerous both to the fetus and to the mother. PET occurs in approximately 12-22% of pregnancies, and is directly responsible for 17.6% of maternal deaths in the United States. We created a structured representation of the PET guideline, which includes the required customizations for applying this guideline in a real clinical setting, in this case, the OB/GYN ward of the Soroka Medical Center.

The objectives of this experiment were to evaluate the feasibility of the Gesher system in structuring a guideline consensus and performing the mark-up process. We were also interested in assessing, in a very preliminary fashion, the clinical editing performance of a senior expert physician and an expert in the guideline’s domain compared to that of an intern with only general medical knowledge. We were also interested in assessing the time and human effort required for completing the specification process, and in discovering common mistakes made during the specification process.

In following section we describe the methodological specification of the PET guideline. Several participants took place in the process. (1) The expert physician and the knowledge engineer together created the ontology-specific consensus that was structured in Gesher. They also created the "Gold Standard" specification in Gesher that was used in the evaluation. (2) The senior OB/GYN expert physician and a general intern (referred to below respectively as the expert and intern) who participated as the clinical editors who used Gesher to create the structured consensus and the structured representation of the guideline. (3) A group of senior expert physicians that took part in creating the clinical consensus based on the recommendations of the American Congress of Obstetricians and Gynecologists (ACOG) guideline for diagnosis and management of preeclampsia and eclampsia.

The structured consensus and the structured representation created by the intern and by the expert were compared to a gold-standard structured consensus and a gold-standard structured representation. The gold-standard representation was created by the expert physician together with the knowledge engineer and was assumed to be the most detailed and correct specification, clinically and semantically. The comparison of the specifications created by the experiment’s participants to the gold-standard was also performed by the expert physician together with the knowledge engineer, using a software tool specifically developed for this purpose. The evaluation tool enabled users to navigate through the complex structure of guidelines which included more than 115 sub-plans, and to compare and grade each part of the structured consensus and the structured representation.

We used objectives measures of completeness and soundness (correctness) in order to measure the level of success of the physicians in completing the specification tasks. The completeness of the specification was defined as the number of knowledge elements (such as sub-plans or plan properties) that exist in the physician-created structured consensus and the structured representation compared to the number of knowledge elements existing in the gold-standard. In order to measure the soundness of the specification products, each of the knowledge elements was assigned a discrete grade describing its correctness; If the content of the element was correct and similar to the gold-standard, the element was graded as “Correct”. If the content was not similar to the gold-standard, the element was assigned one of the following grades; “Not correct, worsening the patient outcome” or “Not correct, not worsening the patient outcome”. The overall level of correctness of a guideline or a group of knowledge-roles is the proportion of knowledge elements assigned with the grade “Correct”.

To evaluate the usability of the Gesher, we used the standard system usability scale (SUS) questionnaire [45] that was presented to the users after performing each of the tasks. To assess the effort in terms of time, we measured the time required for the expert and the intern to complete the specification process.

The objectives of the following experiment were to evaluate our collaborative methodology and the knowledge map interface in terms of the quality of the specification product and the level of usability of the user interface. In addition to assessing the time needed to complete the specification tasks, we were also interested in identifying subjective insights and challenges that arise when representing guideline knowledge.

To accomplish these objectives we designed the following experiment. We asked three knowledge engineers, three physicians, and three combined teams (a physician working together with an engineer) to complete, after receiving a short training in the knowledge map tool, the declarative knowledge specification of the PET guideline. We then asked them to answer a SUS questionnaire. The results of the user's specifications were compared to a gold standard specification that a senior expert physician and a knowledge engineer created.

The training session included some background on the research and its objectives; a presentation of the theory behind the semi-formal representation of declarative knowledge; an explanation about the PET guideline; and an explanation about the knowledge map and the specification task. The training session also included a demonstration of the knowledge map tool and an explanation of the functions and of the display of the interface, which covered the various options available for the user when working with the knowledge map interface. In addition to the explanation, to improve user training, each participant used the tool in a self-training session to gain practice in specifying several medical concepts that were previously demonstrated. It was important to let the users work with the graphical tool and to complete the training session. Feedback from the participants indicated that they found the self-training session to be helpful. At the end of each training session, the participants completed a short questionnaire to determine if the training had been successful and to ensure that they all had the knowledge needed to complete the specification task. The questionnaire included ten multiple-choice questions about the main subjects of the training. After answering each question, the participants were told whether it was the correct answer and mistakes were corrected.

After the training, each participant was asked to create a full semi-formal specification of all of the knowledge concepts from the PET guideline. The guideline includes twenty abstract concepts that were defined, by a senior medical expert and a knowledge engineer during the consensus formation phase on the basis of the PET guideline. Each of these concepts was defined with a textual description (i.e. the structured text representation), and the task of the users was to achieve the most detailed semi-formal representation. To achieve the semi-formal representation, the users used the knowledge map interface to add additional concepts according to the textual definition; to assign them the correct properties (such as the data type of the concept); and to specify the correct "abstracted-from" relations which included specification of value and of temporal constraints. It was pointed out to the users that they should try to avoid redundancy in the knowledge specification i.e., primitive concepts should not appear more than once in the guideline's knowledge description. Redundancy was to be avoided to eliminate misunderstanding by the knowledge engineers when creating, at the next step of the specification, the formal representation. The participants were provided with access to the Gesher system and asked to complete the specification task in their own time. Most participants completed their work in several separate sessions; the time for completing the tasks was measured. For assessing the knowledge map tool's usability, the participant answered a SUS questionnaire, after completing the specification.

In the beginning of the specification, the participants were given a textual description of twenty abstract concepts of the PET guideline knowledge. "Severe PreEclampsia" or "HELLP syndrome", are two examples of these abstract concepts. The abstract concepts included one or more sub-concepts, which were either raw-data concepts or were themselves abstracted from lower level concepts.

To evaluate the completeness of the specification of each of these concepts and to decide on the sub-elements that should exist for each abstract concept, we used a gold standard specification created by a senior physician and a knowledge engineer. Each specification was given a completeness score based on calculating the proportion of the number of elements existing in the specification compared to the number of elements in the gold standard. We examined 57 sub-elements to evaluate the completeness of each participant's specification; thus, the nine participating individuals or teams examined in total 513 (57x9) elements.

To evaluate the correctness of each of the abstract concepts, we examined the gold standard specification and decided on the elements that would be used to measure the correctness. These elements were based on the properties required for creating the semi-formal specification. Examples of these properties included constraints on parameter values, logical operators and parameter data-types (e.g. selection of "Numeric" or "Boolean" data type for a parameter). In order to evaluate the correctness of each specification, we marked each element as "Correct" or "Not Correct", and calculated the correctness as the proportion of elements that were marked "Correct". We examined 221 elements to evaluate the correctness of each participant's specification. (The nine participating individuals and joint teams handled an overall number of 1989 elements ).

In this experiment we created a specification for one aspect of the congestive heart failure (CHF) guideline, managing the Coumadin treatment. We then asked several participants to use the exploration module. The evaluation was conducted with 12 participants: physicians, nurses and students. After being presented with a short demonstration of the tool, they were then asked to use the graphic interface and to answer four short questions regarding the guideline (e.g., what is the Coumadin dose in the loading stage for patients with low BMI levels). After completing the tasks, the participants completed the SUS questionnaire to evaluate the usability of the module.

The overall mean time for training the users was 4 minutes and 20 seconds. The mean time to answer each question was 1 minute and 40 seconds; the mean score for the users' responses to the questions was 85.6%; the mean usability score for the system was 71 (according the SUS scale).