Autism Robotic Theater: a technological tool to contribute to the social interaction of children diagnosed with autism spectrum disorder

According to the World Health Organization, 0.62% of the world’s population has autism spectrum disorder (ASD), which is equivalent to 1 in 160 children. People with ASD have difficulties in communicative behaviors and social interaction. The assistance requirements to improve these deficits are complex. This research ́s primary objective is to propose and implement a robotic theater called ART (Autism Robotic Theater) as a technological assistance tool to contribute to increase social interaction in children diagnosed with ASD. This robotic theater is based on a set of homogeneous humanoid robots performing gestures and actions. ART is formed by 3 NAO robots, a curtain, a narrator, and a software program that allows creating scripts easily and intuitively, to later interpret them. ART was presented in different sessions, to a group of children with ASD under two scenarios. The first scenario aims to encourage the interest and interaction of children with the tool. The second scenario uses the tool to develop a play where robots and children participate, encouraging interaction between them. From the results of an initial and a final evaluation of the children’s performance stands out an increase of 43.75% in the children’s imitation level and, increase of 39.58% in the children’s response and interaction with the robots. Despite the short evaluation time due to the pandemic, the results of the proposed structure have been favorable in the execution of the play in parallel with homogeneous robots and, the impact that the two scenarios have presented in children with TEA.


Introduction
Autism spectrum disorder (ASD) is an increasingly common condition worldwide. According WHO, 1 in 160 children has ASD, and this can change between investigations and regions [1]. Autism is described as an invasive developmental disorder, causing communication and social difficulties [2]. There is no cure for ASD, but several methods and therapies have been developed to work in these deficits. Therapies help to improve patient's social skills, since ASD affects behavior, communication, and language [3], which are need to be treated at young ages. With technology progress several applications have been developed for the treatment of people with autism [4] [5] [6]. These technologies belong to the field of Assistive Technology. In the treatment of children with ASD, the use of technology has proven to be efficient, since it offers a controllable environment, allows for multisensory stimulation, promotes attention, among others [7]. Several works with assistive technology can be found in which different technological tools are used such as tablet and cell phone application: "PictoAprende" [8], "Mi amigo Ben", "Programa Pauta", "El Peapo," among others [7]. There are also advanced tools, that use virtual reality [9] or machine learning for detection and evaluation of autism in patients [10], [11]. Robotics has been gaining influence in the field of therapy for children with ASD due to positive responses produced [12]. Different types of robots have been developed and used to interact with children with ASD. These robots are used in: generation of gestures and movements [13], the recognition of some characteristics in the patient [14], [15], play some interactive games [16], among others. The use of social robotics has also been implemented in ASD patients. In this field NAO robot is widely used. This robot has been a useful due to its high capacity of interaction with the environment and all types of public, its naturalness, and its highly versatile programmability [17]. This set of characteristics has allowed the projects developed on this robot to obtain a higher level of independence, generating therapies and treatments in which the patient with ASD and the robot can interact almost without the presence of a human therapist [18] [19] [20]. The use of more than one robot involved is notably improving specific characteristics in patients such as multiple interactions or joint attention [21], [22]. Also, the use of humanoid robots has been applied in other fields, such as in theater plays [23], [24]. Robotic theaters developed so far are being used as teaching tools for regular school children in STEM (Science-Technology-Engineering-Mathematics) education [25] [26] [27]. However, their use in children with ASD has not been explored. Studies have been conducted on the use of traditional theater in children with ASD showing good results as follows: • Increasing interaction with other children and improvement in face perception and reduction of stress [28]. • Substantial improvement in communication and inter-action with others [29].
• Improvement in general learning in children with ASD or intellectual disability [30].
Taking these results into account, it is interesting to com-bine theater therapy with the use of robotics. This project focuses on the use of social robotics in the area of assistance, teaching and therapy for children and teenagers with ASD, with the use of a robotic theater made up of robotic actors, a narrator and a curtain, controlled by software that allows the user to create and interpret scripts. The structure of the theater is described in more detail in section 2.2. The document has been structured as follows: Section 2 presents system development. Section 3 presents the experimental procedure in which the system applied. Section 4 presents the results obtained. Finally, section 5 presents the conclusions of the work.

Materials and methods
This section presents the social and technical parameters considered for the project, and the design and structure of the tool developed based on these considerations.

Therapeutic and Technical Requirements
Your ART (Autism Robotic Theater) is a technological tool aimed at children with ASD. For the development of the project, the following requirements and suggestions gathered in the Foundation which works with children with autist, were considered: • The striking and not very complex appearance • Avoid large stimuli • Simple operation • Prevent the child from being exposed to cables and other electronic components.
• Avoid sudden movements in robots • Application developed in Spanish • Use of a set of robots so that by way of example, social interaction is fostered.
• Interaction between robots and children in real time.
In response to these requirements, the robotics theater proposal is composed of a group of 3 NAO robots as actors for which a group of gestures and actions was developed following the guidelines of simple operation to avoid causing shocks in children. Also, the theater is composed of a narrator and an automatic curtain as part of the set design. All these elements act automatically and coordinate each other by the use of MQTT (Message Queue Telemetry Transport) communication protocol, which allows for a subscription/publication-based message transport in a simple and extremely light way. The protocol will let the handling of all the elements of the theater in real time. Figure 1, shows the conceptual diagram proposed for ART. This structure let the therapists or general users to use theater's tools following a simple, intuitive, and understandable procedure to create a story. In this way, to fulfill a specific therapeutical goal, the therapist will be able to use this technology, without the help of an engineer. The process of creating a theater play is divided into two parts: story management and story execution. The story management is done offline since it does not require a connection with the robotic theater. In contrast, the story execution is done online since the continuous connection with the robotic actors is required to supervise their actions.

Story Edition Management (off-line)
This module deals with the creation or edition of the story to be interpreted by ART, called script. The script used by the robotic theater follows the structure shown in Figure 2, in analogy to the traditional theater script [31].
The implemented participants so far are described below: • Actors: Humanoid robots with the ability to execute specific actions or gestures.
• Narrator: Participant in charge of storytelling only. A robot or computer can execute it.
• Curtain: Automatic theatre scenery that will indicate if the story or a scene begins or ends through its closing or opening. Script is divided into acts, each act into scenes, and each scene into tasks. All these elements are described below.
• Act: An act, as in a traditional theater script, is each of the parts into which a script is divided. The act is a specific story with a particular purpose or message [31]. An act is made up of one or more scenes, which will execute sequentially to tell the story established. • Scene: The scene is a part of the act, which is made up of one or more tasks. The tasks contained inside a scene will be executed in parallel, whereas each scene will be executed sequentially after all the tasks of the previous scene have been completed. The tool is flexible. Depending on the therapeutic objective to achieve, a structure can be proposed to create short stories, consisting of a single act, with few scenes and tasks. Otherwise, users can create more elaborate stories, made up of multiple acts, with a large number of scenes and tasks. Not all these fields are needed to define a task; this will depend on the character performing the task. Table I shows a summary of the information required for a specific task depending on the character.
If the narrator performs the task, only the dialog is specified. Moreover, if the curtain should act, the "open or close" motion is programmed in the "action" field.
When an actor performs the task, there are 2 cases. The first one is when the selected action is "Talk"; then, only the dialogue specified. In the second case an action is specified, then the execution time is specified. All the actions available to be performed by the actors are shown in Figure 3.

Execution (on-line):
This section explains the processes of resource verification, script interpretation, and script monitoring.

Resource verification
The resource verification process (shown in Figure 4.) validates the participants' requirements to start the script interpretation so that they can operate correctly.
The folder that contains the script, is initially processed to verify the story organization distributed in acts, scenes, and tasks. Then, the configuration of the participants is made, specifying which robot corresponds to each actor role, as well as the IP address and port of each actor robot. Also, voice can optionally be configured for actors and narrator. The interface for performing the resource verification process is described in section 2.3.
Finally, a connectivity check is made with each of the robotic participants, verifying its initial state, detailed in Table 2. As a result of this check, the participants send an internal signal to the system and the user indicating that they are ready for the interpretation

Script interpretation
After checking each theater resource needed for the interpretation of a specific script, the play can start. The MQTT protocol is used for communication between the different participants of ART. Mosquitto broker is used in this project [32]. The use of the protocol makes the theater highly scalable in future works and allows for more participants to be easily integrated. The script interpretation process is carried out in parallel with the monitoring process. The structure of these two processes is shown in Figure 5.
A script called "Story Teller" is the story manager, which will process the information obtained from the script and the configuration of the participants. The story manager will read each specified task, and transform that information into the individual instructions that will be sent to each participant through the MQTT server [32] using the topics described in Table 3.  Where "no" is the number of the robot To communicate instructions from the task to the robot actors, the story manager publishes a message in the topics teatro/actores/nao/uno, teatro/actores/nao/dos and teatro/actores/nao/tres for each robot; to communicate with the narrator, the topic teatro/narrador/relato is used; and to handle the curtain, the topic teatro/telon is used. The general structure used to communicate instructions follows the form: Action /Dialogue/Execution-Time, sending the necessary information (described in Table 1) according to each participant.
The way the topics shown in Table 3 are used, is described in more detail next (in 2.2.2.3).

Monitoring
The script interpreted is also monitored at the same time. To do this, the story manager receives information from each of the participants through the topics teatro/monitoreo and teatro/mensajes. The information received will allow the manager to continuously validate the correct functioning of each participant, and therefore the general progress of the play.
The process of interpretation and monitoring follows the steps shown in Figure 6 and are described below. Figure 6. Phases for the process of interpretation and monitoring • Setup: In phase 1, the story manager obtains the information for a particular task, generates the individual instruction, and sends it through the MQTT server to the corresponding participant. • Validation: In phase 2, the participant receives the instruction and immediately generates status information., which consists of a confirmation packet of having received a valid instruction. For the robotic actors, the battery level and the temperature of their motors are also included. This information is sent back to the story manager and processed to verify that the participant can execute the instruction.
• Execution: If the validation was successful, the participant is in a position to act, so the instruction received in phase 1 is executed. Otherwise, the interpretation ends, and a message is displayed, indicating that the participant was not able to act. • Registration: Finally, when the participant has completed the instruction, it sends another message to the storyteller informing that the task has finished. The storyteller proceeds to register that event, and the task is completed correctly; this allows for constant monitoring of the script.
The entire interpretation and monitoring process is executed for each task specified in the script, following the order set by the scenes.

Implementation of Art
All the robotic theater functionalities described in section 2.2 are done via a user interface. The home screen is shown in Figure 7. By selecting option A1, the program will return to this home screen. To exit the program, the user may use option A2. By using this GUI, users can manage the script by selecting option B1; or interpret a script by selecting option B2, through the options presented at the top of the GUI, or the buttons at the bottom. When option B1 is selected, the interface shown in Figure 8 will be displayed, and if the user selects option B2, the interface shown in Figure  10 will be displayed.

Story Edition Management (off-line)
The process of managing the script is done through the creation of tasks, scenes and acts, following the script structure shown in section 2.2.1. For this purpose, the GUI shown in Figure 8 is presented to the user.
With the button "C", the user can add acts to the script, which will appear in section "c" of the information panel. By selecting the "D" button, new scenes can be added inside an act, which will appear in the information panel as shown in section "d". With the buttons "E" and "F", new tasks can be added or modified respectively, which will be displayed in the information panel following the numbering of the scene to which they belong, as shown in the "e" sections. By selecting the E or F buttons, the interface shown in Figure 9 will be displayed, which will allow the user to enter character, action, dialogue and execution time information in the I, J K and L fields respectively, to define the task. User can save the information about the task or cancel it by pressing the buttons in the field "M". Finally, with the buttons in the G field (in Figure 8), the user can delete acts, tasks or scenes as required by the user.
With the buttons of the "H" field, it will be possible to save the created script or load an existing script to modify it.

Execution (on-line)
Once the script is created, the user can start with the execution process. The GUI shown in Figure 10 allows users to execute the interpretation of the previously generated script and monitoring its progress. Figure 10. Interface for script execution By selecting the button "N," the user will choose the script, whose file path will appear in the "n" field. The configuration for participants is done by any configuration buttons "O", located below the "n" field, or at the top of the interface, shown in Figure 11.
The play configuration is made by entering the corresponding information, participant by participant. The character to be configured will be selected in the field "U". Figure 11. Interface for character configuration Then, the user will have to fill in the necessary fields for each participant: • For actors, the IP address and port are required in the field "V" and voice configuration in the field "W".
• For the narrator, only the voice setting in the "W" field is required.
• For the Curtain, no configuration is required After completing the information for a specific participant, the user will save the information by pressing the "X" button; and then return to the "U" field to select the other participants and set them up. Once all the participants of the script have been configured, the user can save this by pressing button Z. Then, the user will return to the previous screen (Figure 10), where the button "P" must be selected, which will perform the connectivity check with each participant. Finally, the user can start the interpretation of the script, which is done an act by act.
To start with the first act of the script, the button "Q" is pressed, and if necessary, the interpretation can be stopped at any time by pressing the button "S". The user can choose to repeat the act with the option "R" or go on to the interpretation of the next act, with the button "Q." The user must start each act until all the acts in the script have been completed.
In "T" section of the GUI, two progress bars are presented. The left bar will indicate the general progress of the script, and the right bar, the progress of the current act. When the overall progress bar (on the left) reaches 100%, then the interpretation of the script is complete.

Preliminary Test
ART was tested using two different experiments: preliminary and field tests. In the preliminary test, the operation and structure of ART were presented to 4 groups of people which are described below. An example is shown in Figures 12 and 13. The results obtained for the four groups were processed statistically through mode, median, standard deviation, and coefficient of variation, and are presented in Tables 5 and 6. A maximum of 25% (0.25) in the coefficient of variation is expected for the validation of responses.  In general, the tool was positively evaluated, since the answer "Strongly Agree -5" was the most selected (mode) in 92% of the cases in group A, 100% of the cases for group C and 84.6% of the cases for group D. For group B, the answer "Slightly Agree -4" was the most selected, with 61% of the cases.
The results of the two groups of engineers validate the structure and technology chosen for the tool. The results of the students validate the flexibility and intuitiveness of the tool. The results of psychopedagogues and engineer with experience assistive technology, validate the tool as assistive technology, so they recommend ART to be applied as a technological assistance tool as well as a teaching tool

Field Test
In the second evaluation, a field test was carried out in Sangolqui-Ecuador, with the pediatric population diagnosed with ASD, who regularly attends the Virgen de la Merced Foundation -FUVIME. FUVIME is an institution that take care of an average of 100 girls, boys, and adolescents with moderate, severe and profound disabilities of an intellectual and physical nature. The foundation covers different ethnic groups belonging to Ecuador, with a socio-economic level between medium and low [34]. The experimental procedure complies with the ethical regulations and is detailed in the sub-sections below.

Subjects
The experimental subjects belong to the pediatric population of FUMIVE, counting four children diagnosed with ASD with a degree between mild and moderate. The children who participated in the study were selected by the foundation's therapists, after signing the informed consent of their parents.

Ethical Considerations
The experimental procedure is carried out according to the recommendations of the endorsement of the ethics committee issued in the approval document 003-020.

Experimentation
The experimentation process consists of two scenarios carried out in different sessions for four weeks, with a duration of 45 min each. Both scenarios are performed under conditions of supervision by their therapists.

Scenario 1
This scenario aims at familiarizing the experimental sub-jects with the robots, to awaken their attention, and introduce them to ART. For this objective, there is interaction with the robot-actors using short scripts with few gestures and actions, so that later the children imitate the movement of robots. Figures 14 and 15 shows some examples from scenario 1. This initial phase is not only carried out in order to awaken children's interest in robots, but it also helps to assign the roles that children will play, based on their performance with the different elements of ART.

Scenario 2
In the second scenario, the mutual interaction of the four children with the robotic theater is carried out through the performance of a theatrical play created by the experts in childcare with extraordinary abilities of the FUVIME foundation. The designed play allows for the interaction of robotic theater elements, children, and therapists. The participants of the developed script are detailed in Table 7. The play, following the script structure shown in section 2.2.1., consists of 14 acts and a total of 69 scenes and 89 tasks. The story told by the play was created to foster the participation and interaction of children, and at the same time, to sensitize and transmit a message of inclusion towards people with special abilities to the public that can see the play. The story is summarized below: Matías, the protagonist, was a child who, because of the way he was raised, became conceited and spoiled. After an incident in a park with a child with a motor disability, his mother decided to take him to a foundation where there were children with special abilities hoping that his interaction with them would change him. However, it did not work. Matías then had a dream where he met robots whom he perceived as superior to him. Matías wanted to play with the robots, and in this, a wolf appeared that wanted to attack him, and the robots abandoned him. Then the children he had met at the foundation appeared, and they did help him, and it was then that Matías valued them, respecting since then those who were apparently "inferior." At the end of the play, a small dance is performed between the robots and all the actors.
As an example, Figure 16 shows the scene of Matías' dream, where the interaction of one of the children with the robots occurs. Figure 17 shows the wolf scene, in which two children interact with ASD and their music teacher. Finally, Figure 18 shows the final scene of the play in which all the characters dance.

Field Test's Results
The level of social interaction of the experimental subjects with the use of the tool is evaluated at the beginning and the end of the process. The evaluation is detailed in Table 8. The child with ASD is attracted to robots 3 The child with ASD mimics the movement of robots.

4
The child with ASD presents a response to the interactions and activities carried out by the robots.

5
The child actively participates in the play alongside robots and other children. 6 The child's general behavior within the theater is appropriate for the situation.   Table 9 shows the progress of each child. The percentage progress and variation average of the four experimental subjects for each question are measured. The open answer questions 7 and 8 were addressed to therapists, to determine the performance of the tool and the possible benefits when used for longer periods. In the responses presented below, the therapists indicated that pro-longed use of the tool has great potential to help the child with ASD in: • Recognizing emotions in the face of various situations in everyday life.
• Strengthening attention and concentration.
• Encouraging the fixation of the gaze when interacting with a person.
• Contributing to the development of social and communication skills.
• Contributing to the development of motor and cognitive skills.
As a result of the field test, the evaluation of ART in the different scenarios allows us to generalize that children show high interest in the tool. The children are attracted to robots, from the first time they use the tool, and this interest is maintained over time since 3 of the four children have a rating of 6 or 7 (7 being the maximum) in questions 1 and 2, both initial and final evaluation. The tool is also very beneficial in promoting imitation as it can see in the results of question 3, which show the most significant progress with an average of 43.75%, and an increase in all individual cases. The progress presented in question 4 also reflects an essential benefit in children, since it reveals that there was an average improvement of 39.58% in the response that children made to the actions of robots, which indicates that the more they use the tool, the more the children respond to it. Analyzing the results obtained in question 5 (which answers the child's level of interaction in the play) and 6 (referring to their behavior during the performance), we can see that they have an average of 37.50% and 54.17% respectively, higher than some previous questions, but the standard deviation is also the highest, which is confirmed when observing the individual results. This reveals that, unlike the first four questions that have approximately similar behavior in the four children, the 5th and 6th questions have different behaviors when dealing with one child or another. This could suggest that the aspects evaluated in these questions depend on the condition of each child. Finally, it can be remarked that ART fosters an interesting interaction scenario between children, therapists, and robots, the latter being a motivator for their participation. The children interacted and played their roles with joy and predisposition to work. Results of the 4 case studies show improvement of the overall performance of each child during their interaction with the tool and their participation in the play.

Conclusions
ART is a robotic theater for autism designed and implemented to work as a technical assistance tool that contributes to fostering social participation. It allows the therapists to the creation of scripts easily and intuitively so that they can create them without the intervention of an engineer. It allows for the script to be adapted to the needs and requirements of each child based on the premise that, although the degree of autism is the same, each child is different and has different needs. The scripts are based on acts, scenes and tasks and interpreted by the NAO robots, a narrator, and the curtain.
The proposed structure has good results in the execution of a play with parallel processing; that is, robots can perform the same or different actions simultaneously or independently, which helps to get an excellent performance of the acting in the robots.
Although the results presented are good, the evaluation period shortened due to the pandemic that has affected the majority of the world population, so we recommend to carry out the study in more extended evaluation periods. ART was made and proved with a group of homogeneous robots, obtaining favorable results. We consider that ART could be proved with heterogeneous robots, and its results could be promising.