In order to evaluate the approach, we run the implementation of the SDM approach on different domains with various numbers of applications. In addition, we carried out two controlled experiments:
The purpose of
the
first experiment
was to
check whether SDM-generated models are valuable, i.e., may help create
correct, complete, and consistent domain model drafts (to be improved by
humans, i.e., domain engineers). In particular, we tried to evaluate
domain
model comprehension, correctness, and completeness when using SDM.
The subjects of this experiment were
third year
undergraduate students at the university of Haifa, Israel who had previous
knowledge or experience in system modeling and specification and took an
advance software engineering course.
The domain was project management.
The tasks included: (1)
Mapping the
SDM-generated domain model terminology to concepts in two human-made domain
models, (2) Grading 9 different aspects in the SDM-generated domain model
(including correctness, completeness, redundancy, and consistency), and (3)
Correcting the SDM domain model to reflect the subjects' knowledge in the
domain.
The forms can be downloaded
here.
The results showed that
1. Above
98% of the participants mapped the structure correctly
2.
Above 70% of the participants succeeded in correctly mapping the behavior
3. The
main
criticism was: too abstract names or names that are not compatible with
their expected roles in the domain
4. The participants graded the SDM-generated model as follows (out of 5):
Correctness – 3.9, Completeness – 3.7, Redundancy – 4.5 (i.e., not
redundant), Consistency – 4.0, Abstraction level – 3.5
The purpose of
the
second experiment
was to
evaluate
SDM-generated domain model comprehension, correctness, and completeness in
comparison to those of ontologies.
The subjects of this experiment were
third year
undergraduate students at Ben-Gurion university of the Negev, Israel who had
previous knowledge or experience in system modeling and specification and
took an advance software engineering course.
The domain was scheduling.
The tasks included:
(1)
Determining whether the SDM-generated elements are relevant to domain and
how, (2) Mapping the SDM-generated model terminology to that of the ontology
and vice versa, and (3)
Grading 7
different aspects in the SDM-generated domain model (including correctness,
completeness, and redundancy).
The forms can be downloaded
here.
The results showed that
1.
75% of the SDM-generated domain elements were evaluated as relevant to the
domain
2. 73% of the participant answers correctly mapped the SDM-generated model
terminology to that of the ontology
3. 65% of the participant answers correctly mapped the ontology terminology
to that of the SDM-generated model
4. The participants graded the SDM model as follows (out of 5): Correctness
– 3.9, Completeness – 3.7, Redundancy – 3.9, Abstraction level – 2.3
