The study yielded several important insights, but none of these came from comparisons of the initial and final writing samples. Some of the problems encountered in this part of the study provide insight into methodology. They suggest better methods for future work that tries to evaluate improvements in creative writing process and in the quality of creative writing. Other components of the study did have important results. Examination of how children used the software and reading what they liked and disliked indicated how creative writing systems for children can be better designed. Analysis of the post-writing questionnaire indicated that a conversational computer character can lead to greater awareness of some aspects of the writing process.
The EddieEdit group's writing quality as measured did not change during the two weeks of the study. A longer-term study, with stricter initial and final evaluations of writing ability, is needed to determine whether conversation and interaction with a character help students imagine an audience and move from conversation to composition. The EddieEdit group clearly learned more about the writing process, however, as their answers to "What does 'revision' mean?" indicate. EddieEdit users knew the answer in five of seven cases, while none of the six StoryStages users knew. This was despite the fact that identical information about the revision process was present in both programs.
The delivery of this information in conversation, by a character, appears to have helped students understand it better. They seemed to engage more with the conversational character than with a simple list. They read and thought about what Eddie had to say, and then typed a reply. Because they were part of a conversation instead of simply the reader of a list, they made more of an effort to understand the text presented. Although students did not appear to actually revise as a result of the conversations they had with Eddie, they learned what the revision process was like from the conversations they had.
This clear positive result offers some hope that a conversational character can be an effective interface for teaching about the writing process, and perhaps easing the transition from speaking to writing as well. Improvement in writing quality did not indicate that such an interface is effective, at least over a two-week period, but the better understanding of revision that EddieEdit users had signals that this interface has some educational potential and deserves further exploration.
How children actually used the software, as discerned by direct observation and analysis of the saved stories with their additional information, provided ideas about how creative writing software might be better designed. The five main insights were regarding use of text as opposed to graphics, formatting ability, general usability, inevitable reappropriation by children of the software's function, and consistency with the actual writing process.
The programs were written in the Macintosh's graphical environment, but commands and buttons were all labeled with text. There were no special graphics within the program. Although there were long series of textual instructions, second and third grade children had no problem learning to use the software. This points out that an underlying assumption of most developers of children's software needs to be discarded. Currently, the term "multimedia" is used interchangeably with "software" in discussing how computers can be used in education, suggesting that a program must employ more than one medium to be real software. As Amy Bruckman, developer of the textual MOOSE Crossing system, points out, "A media type should be chosen by analyzing the unique requirements of each design situation. ... The benefits of text need to be explained to people seeing it for the first time; graphics are more immediately accepted." (Bruckman 1997) If the software being developed is for writing, it makes sense to use text extensively, perhaps even exclusively. Substituting pictures of bugs for menu titles, as is done in one popular children's writing program, is cute but does not enhance the usability of the software or its suitability as a writing environment.
The issue of how much formatting ability a children's writing program should have is complex. Whole-language advocates would approve of programs that employ drawing, extensive font control, and numerous other formatting functions. These features would allow the computer to be a medium for the sort of writing children do at other points in the classroom, since this educational approach encourages children to think of expressive scribbling and drawing as forms of writing. Even if one does not hold with this view, it is important to consider formatting ability since layout and text style features are very important to children. Formatting can be a distraction from the fundamental writing process used by adult writers, which is about putting words one after another, not about page layout and font selection. Still, some children use the formatting features of the software extensively for useful purposes. For example, they make the text larger to make it more readable, and examine the on-screen pagination to gauge how much they have written. Some of the children who employed the formatting features the most and complained most vehemently about their absence from the three programs in the study also wrote lengthy, high-quality stories. Others played with these features rather than writing.
It is important to give children good tools for writing and to give them reasonable and appropriate control over the appearance of the text. Designers should be careful to make sure all formatting features are included to help the writing activity, however. For instance, there is a compromise between not allowing children to control the size of the font at all and giving them full text formatting ability. The program might let users raise the font for the entire document, but not provide a function for formatting individual bits of text. This would allow users to make their work easier to see while not distracting them with excess formatting ability. This particular feature may not be an ideal one, but it demonstrates how designers might find a middle road between a full-featured page-layout program and a stripped-down text editor. In deciding what formatting functions should be included, developers must consider the overall language-learning philosophy to which they adhere, the focus on the writing task, and the desirability of these features to children.
Ease of use is essential to the success of software for creative writing. Children are very good at figuring out how hard-to-use software works, but the difficulties they encounter along the way can overwhelm the educational benefit of additional features. Commercial programs to help children write have few innovative features but have been designed very well and are well-tested. Their ease of use can be a bigger help to young writers than an innovative educational intervention that is fundamentally sound but poorly implemented.
Observation of students in this study reaffirmed that children will always find new uses for software, explore hidden "features" that adults would never think to use, and generally reappropriate the program for their own purposes. A story-writing program will be used to write things other than stories, no matter how well it conforms to the story-writing task and no matter how hard developers try to keep it from having other uses. The best a software developer can do is keep this in mind when creating the program, and make sure that the program is stable and suitable for its main purpose.
Finally, the original sequential plan/write/revise model did not adequately capture the flow of the writing process. Even when learning about the writing process, writers do not finish all of their planning before writing a word of their story. They engage in editing and revision behavior while they are writing instead of waiting until afterwards. Children found ways to cope with this inadequacy of the programs in the study, circumventing the usual progression by starting over or continuing to write after they had gone through the revision stage. Fortunately, the program was flexible enough so this deficiency in its model of the writing process was not an insurmountable barrier to effective writing. In general, irreversible progression through phases is a bad idea in creative writing software. The improvements to EddieEdit and StoryStages were made with this in mind.
There is a fundamental difficulty in evaluating the writing quality of a group of students before and after an educational intervention. If the two evaluations are separated by a short time, as in this two-week study, there may be little discernible improvement. This problem is worsened in the case of story writing, where children can write about any topic and do not benefit from quick organizational lessons that may help essay writers. If the evaluations are months or a year apart, however, factors extrinsic to the educational intervention may overwhelm its effect. A suitable time frame for evaluating creative writing improvement would probably be more than a month and less than a year. The longer time would also allow children to use the software often - perhaps two or three days a week - but not have to be subjected to daily use, which can be tiresome. Less frequent use over a longer period of time could make for a better study, one which yields clearer results and maintains the motivation of the students who are writing.
The evaluation of writing ability should be done under identical conditions in a more formal atmosphere. The enthusiasm of the students was great during the first writing session since they were beginning the study. During the final session this enthusiasm had been almost completely dulled, and students did not want to write any more. It would help to separate both the initial and final writing sessions from the main part of the study by at least a week. The sessions should not be linked to anything that might cause excitement or resentment. It would also improve results if a more formal testing atmosphere were established in which teachers did not provide help and students did not distract each other. Despite the dampening effect this may have on some students' creativity, it is necessary so evaluators can produce two measures of writing ability that can be compared. The initial and final evaluations should ideally each consist of more than one writing session in which students each produce more than one story.
Relatively unconstrained evaluation by story experts seems to be a fairly good method of determining writing quality, since two evaluators working independently came up with scores that were rather highly correlated. The method could be improved upon by having the experts consult each other and agree on criteria independent from the educational intervention being studied. Although the story experts who evaluated writing samples in this study had different criteria, they shared many of the same principles. It seems likely that they could agree on criteria and apply those criteria fairly consistently to story writing samples. The evaluators also probably had difficulty evaluating unfinished stories. The presence of resolution and an ending are important features of a story mentioned in their criteria. If students were given more time and told to write a story from start to finish, evaluation would be easier and more accurate. Length in words has been used to measure writing ability in prior studies. Even without the formulation of shared criteria, evaluation by experts appears to be a better measure of creative writing quality than length in words.
Two Macintosh programs and a Web version of Eddie's planning conversation resulted from the work described in this paper. The main point was to demonstrate the viability of a conversational computer character to aid in creative writing, thus encouraging additional research into this type of educational intervention. This study should also encourage other developers to integrate conversational characters into software that has been designed with the writing process in mind. EddieEdit, however, already embodies a simple but functional character and can be of use now to educators and students.
StoryStages is a functional program that allows children to plan and revise as well as write. It is personality-free and does not simulate conversation, so it does not provide the educational benefits of a computer character. It does prompt children to think about planning and revising as well as writing, although not as effectively as EddieEdit. Still, StoryStages may be a good program for those who want the benefits of these prompts but do not wish, for whatever reason, to use EddieEdit.
EddieEdit is the first story writing program for children to employ a conversational computer character who helps with the planning and revision processes in story writing. The character Eddie is not the most responsive, knowledgeable or detailed, but he is a character and can help students by conversing with them. Although the author hopes that more advanced characters will be developed soon, EddieEdit can benefit young writers now by increasing their awareness of the writing process, since it is currently available for classroom and home use. The program should also serve as a useful example of how even a simple computer character can effectively converse with children about writing.
The Web version of Eddie can be used by anyone with Internet access and a browser. This version, which offers only the planning conversation and is not integrated with a word processor, is mainly intended to show other developers, researchers, and educators what Eddie is like and how he interacts. Although the Web version has limitations, accessing Eddie on the Web is quicker than downloading EddieEdit. Even though access to the Web version from school will not be widespread, a few young writers who use the Web at school or home might find this version of Eddie useful.
Clearly a longer-term study with thorough initial and final evaluations of writing ability would be of use. Further development of conversational character systems is warranted even before such a study is conducted, however. The final version of EddieEdit corrects many of the usability problems of the earlier version, but Eddie himself could be made more responsive and further interface refinements could be made. Additional development of conversational character features and of the interfaces of both a comparison system and a character system should be undertaken. The conversational abilities could be extended without adding "intelligence" by employing knowledge representation and in-depth semantic understanding. The conversation system could use better pattern matching to determine when the user is giving a "not applicable" reply. The conversation engine could also detect short, repeated replies (such as "ok") that might indicate lack of interest in the conversation. Employing limited "intelligence" in the form of semantic and syntactical analysis could also be beneficial, if such improved linguistic ability on the part of the character is actually used to enhance communication and aid the user in planning and revision. A system incorporating improvements like these would make for a larger potential difference between user groups and be more likely to produce results with significance. After improved software has been developed, long-term studies with more students can help to determine whether a conversational computer character helps students improve more than their peers who do not use such a character.
Additional development should also focus on how to effectively communicate the characteristics of the conversational character. Although Eddie was a fairly well-defined personality as text-based computer characters go, children did not get a good impression of who he was. The icon of Eddie alone may focus them on Eddie's character and invite them to learn more about who he is, and new programs should employ minimal graphics of this sort when appropriate. In EddieEdit, too much discussion of who Eddie is could distract from the story planning task. Eddie's traits and personality should have been more effectively interwoven into the conversation and the advice he gives, so students will realize he is not just a "story helper" but also a kid editor. New programs should be attentive to character development and to effective communication of the character that is developed.
The conversational abilities and representation of character could be certainly be enhanced by having the character manifest itself and converse in additional media, such as sound and animation. This author thinks that, just as drawing software should use mainly graphical media, writing software should use mainly text. Still, it may be possible to enhance the central, textual task by giving a character facial expressions, gaze behavior, and gestures, either to represent emotion or to support conversation with more subtle, foundational behaviors. To fully explore the possibilities of such a character, researchers should create software using varying amounts of animation. Studies should be undertaken to compare the specific effects of different sorts of characters (for instance, animated and non-animated, speaking and non-speaking) on the particular activity of elementary school creative writing.
Eddie is but one type of character who discusses writing in one way, talking about specific story elements. Other characters should be developed that have different personalities and talk about different things. A different character might ask the student for help in writing a story, reversing the relationship. Another character might ask the student only about plot, or only about a topic, or prompt the writer only to visualize or only to free-associate. One character might even be a troublemaker, delivering quick and appropriate replies to the short answers and violent topics that some elementary students offered to Eddie. Asking about story elements is traditional, but it is certainly not the only approach. A character who is upbeat and knowledgeable is also typical, but other sorts of character may be more engaging and effective. Ideally, students should be given a choice of what type of character they wish to talk to and what type of discussion they wish to have.
Another important consideration is how a conversational computer character should be integrated into software that uses several effective educational interventions. A naive computer character that does not examine the story the user has written could exist alongside grammar-checking, style-checking, and spell-checking tools. There should, however, be special consideration of how such features might be best integrated. Some may find it better to educate the computer character and have students use that character as a uniform planning, revising, and editing interface.
Finally, developers and educators need to consider together how the use of a conversational computer character might fit into their own educational philosophy and teaching strategy. Those who see the computer as more of a tutor might consider developing characters that not only stimulate thinking but also directly instruct. Others might create characters who suggest topics for stories, rather than just asking about structure and story elements. Although such approaches were avoided here because of the author's educational goals and beliefs, the effective intervention of a conversational computer character can be employed by others who do not share the same overall outlook. Even those who agree with this outlook need to consider how a conversational computer character can work with other parts of the language arts curriculum. The character could, for instance, discuss reading as well as writing, giving examples of effective use of story elements (if they are the topic of the planning discussion) in some of the user's favorite stories. A conversational computer character can - with additional research, further work from educational software developers, and the support of teachers - play a meaningful role in elementary creative writing education and make a difference in the language arts classroom.