Mathematical teaching and learning materials like schoolbooks include, especially for the acquisition of elementary computational competencies in primary and early secondary school, practical examples and explanations on how to stepwise proceed when solving mathematical problems. These instructional parts contain implicit visual didactic information where the problem-solving processes are explained using pure visual methods (e.g. graphical and pictorial cues, spatial arrangements for sequencing or establishing relations, color-coding, and other layout-based content representations). So far, there is no semantic markup or workaround for non-visual user interfaces such as refreshable braille displays or text to speech (TTS) engines to represent this implicit visual information, what considerably affects teaching and learning of blind children. Furthermore, this absence of efficient explicit semantic markup of these implicit didactic elements is one key reason for the difficulties faced by the target group in STEM education and related job opportunities. The core objectives of this work are: (1) to research and develop novel strategies to support blind students by marking up visual didactic information used in mathematical schoolbooks, (2) to evaluate if, what, and how much support is given by editor-based functionalities providing a non-visual alternative to implicit elements, (3) to study more efficient editor-supported stepwise procedures for “doing math” and finally, (4) if all this supports the development of a better mental model of the calculation process.
With our model, implicit mathematical content does not only become perceivable but computable. This structured format – together with specific tools – allows real-time synchronization of different presentation modes of math (visual, braille, audio). This thesis outlines how our model facilitates better communication and hands-on interaction to foster the educational and vocational inclusion. The thesis starts with analytical and conceptual research on developing an approach for making the implicit didactic information behind pure visual presentation techniques accessible. A framework for semantically marking up the implicit visual elements used in mathematical schoolbooks going beyond linear text is presented. These conceptual approaches are accompanied by standardized empirical methods (desktop research; expert interviews; qualitative, partially structured interviews and participatory observation) of teaching techniques used in the classrooms for blind students. Both the conceptual and empirical results are used to define the markup for visual and dynamic content. In a second step, the semantically marked-up and enriched content forms the basis for the prototype of the Interactive Didactic Math Inclusion Working Environment for Blind Students (IDMILE). IDMILE is specified and defined as an interactive, dialog-based assistant that guides a blind student through the calculation process. It also provides supportive functionalities for a better management of dynamic, parallel procedures. The process to display dynamic, parallel procedures and information is highly complex because the genuine modalities in use (audio, refreshable braille display) present information in a (limited and limiting) sequential way when it comes to parallel events.
The prototype supports MathML Elementary Math representation, which facilitates better communication between sighted and blind users. Currently, the IDMILE covers four basic arithmetic operations: addition, subtraction, multiplication, and division. IDMILE is implemented by strictly following accessibility requirements for the target group and their Assistive Technology (braille, audio) using the Eclipse SWT framework and Microsoft UI Automation. In a third and final step, the thesis does an empirical analysis if and how the IDMILE approach impacts on doing math by blind students in terms of navigating through the spatial construct of mathematical expressions and obtaining a mental model for building up the understanding and procedural problem-solving skills. In addition, a usability evaluation was conducted using heuristic evaluation method and an accessibility evaluation was done using Microsoft’s accessibility testing tools. The thesis proves that the IDMILE prototype (based on explicit as well as semantically marked-up implicit, pure visual didactic information embedded in mathematical schoolbooks) allows blind students a) to faster learn and understand basic math, b) to solve calculation tasks more efficiently and c) to better communicate and interact with sighted teachers and peers. Finally, the thesis outlines future work on how the semantically enriched markup and interfacing computational tools could lead to new approaches in doing math by as well as for and with blind students.