• Home
• About
• Contact
• Site Map
• Search CT4ME

Math Topics

• Common Core
• Initiatives
• Methodology
• Resources
• Projects
• Manipulatives
• Software

Learning Support

• Standardized Test Prep
• Technology Integration
• Assisting Readers

Professional

• Associations
• Journals
• News
• Professional Development
• Education Standards
• Education Research
• CT4ME Publications

Technology Integration:
Essential Questions (Page 2 of 3)

 

How should technology be used?

This question can be addressed from two perspectives--what we desire for all students, teachers, and providers of education in general and then specific to mathematics.

Technology Use in General

Various organizations, teachers and students themselves have indicated ways that technology should be used.  While the words might differ, there are commonalities among those recommendations.  But, the overall reason for using technology in instruction is to reach our ultimate goal as educators, which should be to enhance the achievement of learners.

Unfortunately, there are Digital Access, Design, and Use Divides noted in the 2024 National Educational Technology Plan (NETP).  For example, in regard to the digital use divide, there is "Inequitable implementation of instructional tasks supported by technology. On one side of this divide are students who are asked to actively use technology in their learning to analyze, build, produce, and create using digital tools, and, on the other, students encountering instructional tasks where they are asked to use technology for passive assignment completion" (U.S. Department of Education, Office of Educational Technology, 2024, p. 9).  The 2024 NETP includes recommendations for closing each divide.

In general, the National Academies of Sciences, Engineering, and Medicine (2018) drew the following two conclusions regarding the use of digital technologies for learning:

1. The decision to use a technology for learning should be based on evidence indicating that the technology has a positive impact in situations that are similar with respect to:

• the types of learning and goals for learning;
• charactistics of the learners;
• the learning environment;
• features of the social and cultural context likely to affect learning; and
• the level of support in using the technology to be provided to learners and educators.

2. Effective use of technologies in formal education and training requires careful planning for implementation that addresses factors known to affect learning.  These factors include alignment of the technology with learning goals, provision of professional development and other supports for instructors and learners, and equitale access to the technology.  Ongoing assessment of student learning and evaluation of implementation are critical to ensuring that a particular use of technology is optimal and to identifying needed improvements. (p. 196)

John McCarthy (2018) proposed four categories for how learning can be supported by digital tools: managed learning, orchestrated learning, collaborative learning, and authentic learning.

• Managed learning involves using management systems and tools to keep track of assignments, grades, announcements, and communications.
• Orchestrated learning is teacher directed.  Per McCarthey,"They structure and provide assignments that students complete to grow academically. The tasks may vary from completing research, virtual lab experiments, posting responses to a discussion board, watching video tutorials, and completing foundational tasks through learning centers or stations."
• Collaborative learning "is based on where students’ skills are currently, then uses digital tools to support their growth individually and/or through the support of a peer team. Such student-centered work enables the teacher to facilitate thinking and coach growth that is personalized to each learner."  Teachers plan the learning experience, but students lead the learning.
• Authentic learning involves public display of learning via such tools as blogs, wikis, school or outside webpages, YouTube or other social media, online discussion boards, and so on.  (McCarthy, 2018)

When considering the types of digital learning available and their range of use, which can be personalized for learners, some "rules of thumb" have emerged.  Per Cheryl Lemke (2014), educators do need to consider:

The Instruction, which ranges from the didactic to coaching and inquiry

The Complexity of the Learning, which ranges from basic skills through higher order thinking

The Level of Authenticity, which ranges from simulated, through authentic, real world experiences. (Lemke, 2014)

In the series of Technology Briefs for NCLB Planners, the Northeast and the Islands Regional Technology Consortium (NEIRTEC, 2002) presented Strategies for Improving Academic Achievement and Teacher Effectiveness:

• Use technology linked to district and school initiatives to support learning in the content areas.
• Integrate technology into the curriculum, rather than making technology a separate subject area.
• Use technology to assist with data-driven decision making.
• Use technology to support different learning styles and to meet the needs of all learners, including those with disabilities.
• Use technology as a vehicle for professional development.

In Maximizing the Impact: "The Pivotal Role of Technology in a 21st Century Education System" (2007), the International Society for Technology in Education (ISTE), The Partnership for 21st Century Skills, and the State Educational Technology Directors Association stated that technology can be used in nine key areas to assist with teaching and learning:

• Building conceptual understanding of core content;
• Addressing misconceptions;
• Fostering inquiry and investigation;
• Applying knowledge and skills to interdisciplinary challenges;
• Creating and transforming knowledge for meaningful purposes;
• Collaborating with others;
• Apprenticing with experts;
• Engaging and motivating students; and
• Differentiating instruction to meet individual needs. (pp. 9-10)

Technology can be used for "information, images, interactions, and inquiry" (Quirk, in Pollock, 2007, p. 102).  To this end, ISTE's (2007) release of National Educational Technology Standards for Students: The Next Generation indicated that to learn effectively and live productively in an increasingly digital world, students should know and be able to use technology for creativity and innovation; communication and collaboration; research and information fluency; critical thinking, problem solving, and decision making; digital citizenship; and technology operations and concepts.  Students should be able to:

• Create original works as a means of personal or group expression.
• Use models and simulations to explore complex systems and issues.
• Interact, collaborate, and publish with peers, experts or others employing a variety of digital environments and media, including globally.
• Locate, organize, analyze, evaluate, synthesize, and ethically use information from a variety of sources and media.
• Plan and conduct research, manage projects, solve problems and make informed decisions using appropriate digital tools and resources.
• Advocate and practice safe, legal, and responsible use of information and technology.
• Troubleshoot systems and applications.

ISTE's 2007 National Educational Technology Standards for Students focused on "using technology to learn."  In 2016 ISTE revised those student standards to focus on "transformative learning with technology."  The seven standards, along with indicators within each, address empowered learners, digital citizenship, students as knowledge constructors, innovative designers, computational thinkers (this latter being a new addition to the standards), creative communicators, and global collaborators.

Teachers also have expectations regarding the digital tools they use for instructional purposes.  Per its survey of a national representative sample of more than 31,000 U.S. public school teachers, the Bill & Melinda Gates Foundation (2014) found that teachers identified six instructional purposes for which digital instructional tools are beneficial.  Hence, one could conclude that the following per those teachers are additional ways that technology can be used:

• Delivering instruction directly to students
• Diagnosing student learning needs
• Varying the delivery method of instruction
• Tailoring the learning experience to meet individual student needs
• Supporting student collaboration and providing interactive experiences
• Fostering independent practice of specific skills (p. 9).

When used for professional development, "technologies as videoconferencing, online learning, networking, and instant messaging can support professional development and professional learning communities.  Using technologies like these, educators can learn and collaborate with peers, mentors, experts and community members routinely. They can build ongoing professional relationships, develop capacity in teaching 21st century skills, benefit from just-in-time communications, and reduce the time and expense of travel" (Maximizing the Impact, 2007, p. 13).

Indeed, teachers need more technological skills to be able to effectively integrate technology into classroom lessons and help students to become "collaborative, problem-solving, creative learners through using ICT," according to the United Nations Educational, Scientific, and Cultural Organization (UNESCO, 2011, p. 3).  In order to form some consensus about those skills, many of which were noted above, and to determine a plan for their acquisition, in 2008 UNESCO and colleagues Cisco, Intel, Microsoft, the International Society for Technology in Education and the Virginia Polytechnic Institute and State University set up the ICT Competency Standards for Teachers project.  Revised in 2011 and then in 2018, the ICT Competency Standards for Teachers (v 3) addresses six aspects of a teacher's professional practice including understanding ICT in education policy, curriculum and assessment, pedagogy, application of digital skills, organization and administration, and teacher professional learning.  The framework is arranged in three stages of a teacher's development: knowledge acquistion (called technology literacy in UNESCO 2011), knowledge deepening, and knowledge creation (UNESCO, 2018, pp. 8-9).  It can serve as a syllabus with detailed descriptions of the specific skills to be acquired by teachers within each skill set/module.  It can serve as a basis for developing professional development programs and teacher education, and as a checklist for skills acquired.

Also from a general perspective, district leaders also need to use technology.  "Technology can support administration in providing instructional leadership, managing learning environments and professional learning communities, and making decisions that support proficiency in 21st century skills. Networking technologies, for example, can support administrators in communicating with staff members, parents and community members. Data management systems enable states, districts and schools to make sense of the mountains of data they collect, monitor technology and other resources, and track trends in student achievement. In this sense, technology is a “data tool for education to better understand and inform educational and instructional decision making” (Maximizing the Impact, 2007, p. 13).

Visit Teaching NOW!, an online television and radio series that investigates the relationships between education and technology.  The series, funded in part by the U.S. Department of Education, explores issues, ideas, and strategies surrounding education and teaching.

 

 

Technology Use In Mathematics

Ted Hasselbring, Alan Lott, and Janet Zydney (2005) noted six purposes of technology use for supporting student mathematical learning and their development of declarative, procedural, and conceptual knowledge:

1. building computational fluency;
2. converting symbols, notations, and text;
3. building conceptual understanding;
4. making calculations and creating mathematical representations;
5. organizing ideas; and
6. building problem solving and reasoning. (p. 2)

Elaborating on those, The Partnership for 21st Century Skills (P21) developed ICT Literacy Maps for core subject areas to illustrate how technology assists with attaining and utilizing 21st century skills.  Representative ways that technology can be used in mathematics at grades 4, 8, and 12 are included.  For example, thoughts derived from the Math ICT Literacy Map:

• Newspapers, books, spreadsheets, graphing programs, calculators, computers, Internet, films, TV programs, Websites, databases, internet and digital libraries can help students gain information and media literacy.  They are sources for the study of data analysis.

• Word processing programs, graphic programs, presentation software, desktop publishing programs can help students gain communication skills.  These are applicable for math projects.

• Word processing software, manipulatives, calculators, graphing calculators, spreadsheet software, probes, GPS, and geometry tool software are useful for developing critical thinking and systems thinking.  Use these tools when problem-solving, keeping journals of mathematical experiences, and creating graphical representations of data, for example.

• Manipulatives, calculators, graphing calculators, Smart Boards, and presentation software help students to develop problem identification, formulation, and solution skills.

• Digital cameras, laptop computers, multimedia presentation software, graphing calculators, probes/CBRs, Website development software can be used to enhance creativity and intellectual curiosity.  For example, students might take photos showing geometry representations in their surroundings and create a math slide show.

• Calculators, computers, newspapers, Internet, spreadsheet programs, presentation software, video equipment can help build interpersonal, self-direction, and collaborative skills.  Students might create portfolios with examples of problem-solving situations in real life, or reflections on their problem-solving and thinking, and their understanding and learning of math concepts.

• Internet, presentation software, word processing, desktop publishing can be used to communicate with students in other communities or countries, participate in national math competitions, or to discuss concepts with outside experts in online bulletin boards. These become tools for accountability and adaptability.

• Internet, presentation software, newspapers can be used for community service projects, and for collecting data to be analyzed with math tools and then making reports on local issues. Such use enhances and develops social responsibility.

The National Council of Teachers of Mathematics (2011) views technology as an essential tool for learning mathematics and acknowledges that teachers and curriculum play a critical role.  Per NCTM:

Technological tools include those that are both content specific and content neutral. In mathematics education, content-specific technologies include computer algebra systems; dynamic geometry environments; interactive applets; handheld computation, data collection, and analysis devices; and computer-based applications. These technologies support students in exploring and identifying mathematical concepts and relationships. Content-neutral technologies include communication and collaboration tools and Web-based digital media, and these technologies increase students’ access to information, ideas, and interactions that can support and enhance sense making, which is central to the process of taking ownership of knowledge. Findings from a number of studies have shown that the strategic use of technological tools can support both the learning of mathematical procedures and skills as well as the development of advanced mathematical proficiencies, such as problem solving, reasoning, and justifying.  (NCTM, 2011, para. 2)

P21 in collaboration with the Mathematical Association of America, NCTM, and dozens of math educators also developed The 21st Century Skills Map for Math (2011), which provides connections between the Common Core State Standards and 21st Century Skills.  Lesson plans, learning outcomes, and suggested tools for integrating the skills are provided with examples for grades 4, 8, and 12.

 

 

Back to top

 

What are courseware and digital content types for mathematics?

In general, technology tools might fall into four broad categories: mathematical action tools, collaboration tools, assessment tools, and communication tools (McCulloch, Hollebrands, Lee, Harrison, & Mutlu, 2018).  The selection of any tool would be determined by its role, the content that is being taught, how it is taught, and a decision on what type of technology would best help in achieving instructional goals.  Ease of use is also a factor in selection for both teachers and students. 

Hubbell and Miller (2013) identified nine content types:

• word processing
• organizing and brainstorming
• data collection and analysis
• communication and collaboration
• instructional media
• multimedia creation
• instructional interactives
• database and reference
• kinesthetic technology (Hubbell & Miller, 2013)

In addition, there are technologies specific for teaching mathematics, which can also enhance development of declarative, procedural, and conceptual knowledge.  Mark Schneiderman (2006) identified the following courseware and digital content types:

1. Tutorials: Programs are used to introduce math concepts and then to provide practice, assessing learners as they progress.  The primary focus is on identification of existing knowledge / formative assessment and acquisition of new information / development of new skill.  The secondary focus is on application of new information / practice of new skill and demonstration of mastery / summative assessment.

2. Skill-Building / Drill & Practice: Unlike tutorials, these programs assume learners have some prior knowledge.  The primary focus is on application of new information / practice of new skill.  The secondary focus is on acquisition of new information / development of new skill and demonstration of mastery / summative assessment.  There are levels of difficulty to meet learner needs, often with hints, explanations, and graphical representations.  Programs are often in game format.

3. Comprehensive Courseware: Programs provide a core curriculum with support for the learning process and might combine tutorials, practice, and assessment.  Skill mastery is tracked; a student data management and reporting system is often included to inform instruction.

4. Problem-Solving: Programs require learners to use specific math skills to solve challenges or puzzles. Focus is on application of new information / practice of new skill and refinement of meta-skills.  Problems presented might have one correct answer and/or one solution path or multiple correct answers and paths.

5. Test Prep: These programs assess knowledge, particularly for standardized test preparation.  The focus is on application of new information / practice of new skill and demonstration of mastery / summative assessment.

6. Simulations & Visualization: Multimedia simulations are often embedded in applications above, and can also be stand-alone.  They can be used to help learners visualize and interactively explore concepts, and apply new conceptual knowledge to real-world situations.  Some video-based simulations are less interactive.  Focus is on acquisition of new information / development of new skill and application of new information / practice of new skill.

7. Educational or Serious Games: Schneiderman (2006) said this "new category of courseware is emerging designed around more authentic gaming concepts. These applications provide more immediate and ongoing feedback, require more concentrated and lengthy attention, allow repeated practice, motivate increased time on task, and employ a very leveled and contextual approach to building skills and knowledge" (p. 11). The primary focus is on acquisition of new information / development of new skill and application of new information / practice of new skill.  Secondary focus on identification of existing knowledge / formative assessment, demonstration of mastery / summative assessment, and refinement of meta-skills.

 

Back to top/

 

What principles should guide your approach for integrating technology into instruction?

According to Marc Prensky (2005), today's students have mastered a variety of tools and "[e]ducating or evaluating students without these tools makes no more sense to them than educating or evaluating a plumber without his or her wrench" (p. 12).  Prensky indicated that their system of communication involves instant messaging, sharing information through blogs, buying and selling on eBay, exchanging through peer-to-peer technology, meeting in 3D worlds, collecting via downloading, coordinating and collaborating through wikis, searching with Google, reporting via their camera phones, programming, socializing in chat rooms, and let us not forget learning via Web surfing.  Their tools are just extensions of their brains.

The use of these new tools is among trends driving our global economy (Anderson, 2006). These tools "harness the wisdom of the crowd," enable "a shared culture of fandom, commentary, and camaraderie" to be developed, and ultimately are taking the Information Age to a new level, which Chris Anderson (2006) calls the "Age of Peer Production" (p. 132).

Although teachers know it's not the medium, but instructional methods that cause learning, some might be tempted to use new media in instruction (e.g., virtual worlds, gaming environments, blogs, wikis, intelligent agents, video files) without a clear plan for an educational outcome.  After all, their students appear to have already mastered many of those, as Prensky noted, and are quick to try out new tools as they become available.  However, technology should not be implemented just for the sake of adopting technology.  It must serve a role in learning.

Key Parameters and Questions to Answer

With all the options for using technology, new hardware, software and app products springing up on the market, and schools adopting 1-to-1 approaches to technology integration, decisions must be made carefully when selecting new technology.  A first step in selection would be to conduct a needs assessment.  The Needs Assessment Guidebook from the State Support Network (Cuiccio & Husby-Slater, 2018) explains the process and provides additional resources.  "The needs assessment process is an important first step in improving the effectiveness of education investments that lead to better outcomes for students" (p. 1).

Selection of Devices, Apps, and Media:

Thornburg (2014) provided three guiding principles for choosing computers for schools.  Consider in order:

1. What is the educational objective?
2. What software meets the objective?
3. What platform(s) run the desired software?

His reasons for why this process is important deal with cost and time.  "First, these tools are expensive and schools need to get as much use out of them as possible.  Second, time in the classroom is scarce, and it needs to be used wisely" (p. 28).

Certainly, a key consideration is that apps and media need to run on multiple devices.  Lucas Johnson (2022) pointed out that "Knowing which software and tools educators plan to use use, and investigating their requirements, can help when choosing devices."  Compatibility among devices and how students will access apps and media are key issues.  For example, consider the hardware's operating system, as it differs in devices such as laptops and tablets.  Some apps can be accessed via a web browser; others need to be downloaded and installed onto a device.  Consider compatibility with the school's existing infrastructure.  There are multiple access issues, such as log-in requirements, access at home versus at school, any need for creating accounts, passwords and usernames and what to do if a learner forgets those.  Johnson noted:

"Tools that don’t require users to log in are potentially easier to use and more accessible. But they are much less useful for teachers for tracking individual progress and assessment.

Tools that require credentials to be accessed should ideally use a single set of credentials or single sign on process.

Not having access to a physical device and broadband internet are also potential impediments." (Compatibility and Access sections)

Vicki Hancock (1992) discussed the LOCATE Model (learners, outcomes, comparison, assembly, trial, and evaluation) for selecting and evaluating instructional media.  According to this model, those who select media should consider the needs of the intended learners, and whether or not the outcomes of instruction require media.  Potential media should be compared for authenticity, suitability, organization, technical quality, and special features.  The assembly component requires gathering and ensuring that all components (e.g., hardware, software, room/environmental considerations, support staff/volunteers) are available so that the media will be totally usable by the learners.  Hancock suggested a trial period before purchase to test the product with learners for their reactions and to determine if the product includes subject matter as intended.  Evaluation should include "an appraisal of the materials themselves and of the methods used to integrate them into learning activities" (para. 9).

It would also seem wise to consider some basic parameters for selecting applications to help prevent students and educators from becoming "app-a-holics" --a term William Tolley (2015) used for mass accumulation of apps that end up overwhelming students and educators.  Tolley noted parameters to help simplify the selection process.

1. Apps should have direct relevance to class goals, and not be redundant (i.e., don't choose two apps that essentially do the same thing).
2. Apps should be flexible, where possible, in terms of being operable cross-platform and cross-device.
3. They should be authentically useful, which means the best apps are also useful to learners outside the school environment, and have longevity. (Tolley, 2015)

Joel Smith and Susan Ambrose (2004, online pp. 1-2) of Carnegie Mellon University posed seven more detailed questions to help educators think in a systematic way about how and when to incorporate any new pedagogical strategy, including media, into instruction. Their fundamental questions included:

1. What is the educational need, problem, or gap for which use of new media might potentially enhance learning?

2. Would the application of new media assess students' prior knowledge and either provide the instructor with relevant information about students' knowledge and skill level or provide help to students in acquiring the necessary prerequisite knowledge and skills if their prior knowledge is weak?

3. Would the use of new media enhance students' organization of information given that organization determines retrieval and flexible use?

4. Would the use of new media actively engage students in purposeful practice that promotes deeper learning so that students focus on underlying principles, theories, models, and processes, and not the superficial features of problems?

5. Would the application of new media provide frequent, timely, and constructive feedback, given that learning requires accurate information on one's misconceptions, misunderstandings, and weaknesses?

6. Would the application of new media help learners develop the proficiency they need to acquire the skills of selective monitoring, evaluating, and adjusting their learning strategies?  Some call these metacognitive skills.

7. Would the use of new media adjust to students' individual differences given that students are increasingly diverse in their educational backgrounds and preferred methods of learning?

Data Collection and Student Privacy:

A primary question deals with vendors' collection of data and how they and districts are ensuring student privacy of data.  There is a real concern among the education community, including teachers, students and parents, that ed tech companies are collecting more student data than is necessary through their use of devices and software in the classroom, often without their or parents knowledge, and storing it indefinitely.  Such data might include personal identifying information, "browsing history, search terms, location data, contact lists, and behavioral information."  In effect, many ed tech companies are spying on students.  Districts that do not have an adequate privacy policy or none at all are "unwittingly helping them to do it," according to results of a 2016 large-scale survey and interview study from the Electronic Frontier Foundation (Alim, Cardozo, Gebhart, Gullo, & Kalia, 2017, p. 5).  Fortunately, the authors provide suggestions for stakeholders in their report, Spying on Students: School-Issued Devices and Student Privacy, to help remedy the problem.

Before purchasing any software/app, examine the vendor's terms of service agreement to ensure strong privacy and data security to protect students.  Edsby.com provided a series of questions to ask vendors about student data privacy, and noted the Common Sense Privacy Program is a good place to start when looking for a solution, as the program rates technologies and provides general information about a product of interest.  Per Lucas Johnson (2022), "It is important to be aware of who has access to the information learners create and to have a clear understanding of who the students can communicate with and how" (Data privacy and security section).

Standards Alignment:

Noel Enyedy's (2014) review of personalized learning revealed a final question to consider in the selection process: Does software align to standards?

Setting aside the controversy surrounding national academic standards, where academic standards are in place educators adopting instruction via technology should insist that developers provide software aligned with the standards. ... Adopters might also consider seeking software that reflects national assessment systems being developed (such as the Smarter Balanced Assessments), so that instructional systems parallel accountability systems and can possibly alleviate some of the onerous and time consuming aspects of testing to the high standards set by the Common Core and Next Generation Science Standards. (p. 22)

If you can answer "yes" to one or more of the above questions when considering using a particular strategy or a new media, then your selection has a chance of making a difference in learning.

Universal Design for Learning and Accessibility:

Principles of universal design should also be considered when selecting media for use in an instructional program.  Universal Design for Learning (UDL) from CAST calls for students to have multiple means of expression, representation, and engagement in their learning.  Materials provide those elements and have scaffolds built in (Deubel, 2003).

The phrase "universal design for learning" occurs multiple times within the Every Student Succeeds Act.  For example, local educational agencies are encouraged to "use technology, consistent with the principles of universal design for learning, to support the learning needs of all students, including children with disabilities and English learners" (114th Congress, 2015, Sec. 4104. State Use of Funds, p. S.1177-172).  UDL is tied to accessibility.

"Accessibility refers to the design of apps, devices, materials, and environments that support and enable access to content and educational activities for all learners. In addition to enabling students with disabilities to use content and participate in activities, the concepts also apply to accommodating the individual learning needs of students, such as English language learners, students in rural communities, or students from economically disadvantaged homes. Technology can support accessibility through embedded assistance—for example, text-to-speech, audio and digital text formats of instructional materials, programs that differentiate instruction, adaptive testing, built-in accommodations, and other assistive technology tools" (U.S. Department of Education, 2017, p. 5).

Educators have a legal responsibility to provide accessible content.  To illustrate, materials include curricula and resources (e.g. digital textbooks, e-books, online videos), devices include smart phones and tablets, digital tools would include computers, apps, and games, and platforms would include online learning platforms and websites (Center on Technology and Disability, 2016, pp. 2, 6).

For students with disabilities (e.g., vision, hearing, learning), technology use may pose unintended barriers to learning.  "[F]eatures such as text-to-speech, speech-to-text, enlarged font sizes, color contrast, dictionaries, and glossaries should be built into educational hardware and software to make learning accessible to everyone" (U.S. Department of Education, 2017, p. 21).  Overcoming barriers also includes providing captioned videos, alt-text to describe images used in websites and e-books, and ensuring content can be read by screen-readers.  This latter is a challenge for charts, graphs, formulas in STEM content (Center on Technology and Disability, 2016, p. 3).

Regular access to Closing the Gap, a Web site devoted to computer technology in special education and rehabilitation, will provide articles, product information, discussion forums, and other resources of value on accessibility.

The Digital Accessibility Toolkit: What Education Leaders Need to Know produced by the Center on Technology and Disability (2017) contains resources, tips, and information to guide educators in taking a proactive approach to accessibility.  The four sections of the toolkit define accessibility and share why this effort is important today, identify the legal requirements for digital accessibility, describe the benefits of digital accessibility, and explain the procurement of accessible technology.

 

Back to top

 

When can you expect technology to be effective?

Several factors have the potential to influence the effectiveness (i.e., success or failure) of an edtech implementation.

The International Society for Technology in Education (ISTE, n.d.) identified seven essential conditions for effective tech use in schools:

1. A shared vision among all stakeholders for transforming learning through the effective use of technology
2. Implementation planning "for building and sustaining technology infrastructure, evaluating and selecting digital learning resources, and providing and sustaining professional learning and coaching."
3. Equitable access by all "to digital devices, internet connectivity, capable teachers and technology platforms required to support the learning vision."
4. Prepared educators who know "how to use technology in appropriate ways, aligned to learning research and the school system’s vision for learning."
5. Skilled and sufficient technical support
6. High quality learning activities and content aligned to standards
7. Ongoing evaluation in partnership with all stakeholders. (ISTE, n.d.)

Evidence of effectiveness is a key factor.  The Dear Colleague Letter: Leveraging Federal Funds for Teaching and Learning with Technology (Office of Educational Technology, 2023) included sets of questions to "help support decision-making related to the critical areas of needs assessment, policy and infrastructure, alignment with instructional approach, professional learning, and evidence" (p. 4).  For example, when determining the level of evidence, consider if it is anecdotal, correlational, or causal.  If a study was conducted, how rigorous was it?  How similar is your context to that of the context of the research study? (pp. 5-6)

According to the EdTech Evidence Exchange (2021), "Edtech decision makers currently select and implement technologies with almost no information about what is likely to work in their schools" (p. 4).  It undertook the challenge to solve the problem in its EdTech Genome Project and hypothesized 10 factors (2020) that influence the success or failure of edtech:

1. The Vision for Teaching and Learning with Technology evidenced by a clear direction, purpose, and rationale for edtech implementation.
2. The Selection Processes, which address consistent methods to identify technologies for evalution, evaluating those prior to purchasing, and then procurement.
3. Competing Priorities, which might include a number of initiatives, and competition between tech and non-tech initiatives.
4. Availability of appropriate Infrastructure and Operations Resources such as hardware, software, internet access, considerations for student home devices and connectivity, tech support, financial resources.
5. Implementation Systems and Processes, which monitor the presence and quality of systematic and inclusive processes a school/district uses to implement edtech tools after procurement and over multiple years.  This includes setting and monitoring usage and engagement goals, as well as evaluating effectiveness.
6. Professional Learning (Development) and Support for users.
7. School (Staff) Culture, including shared beliefs, values, and assumptions.
8. Strategic Leadership Support from School and District Administration for users who are implementing edtech tools.
9. Teacher Agency/Autonomy, including the extent to which teachers are included in edtech decision-making for adoption and implementation.
10. Teacher Beliefs and Knowledge about technology and their understanding of curriculum, instruction, and assessment.

The EdTech Evidence Exchange released the final report of the EdTech Genome Project in July 2021.  In this project "education technology [edtech] only include[d] technologies used in instruction, as opposed to technologies that purely support educator workflow (e.g., an online teacher evaluation instrument)" (p. 119).  The following are among key takeaways:

• "Harnessing the expertise of a broad and diverse group of practicing educators, researchers, industry representatives, policy makers, and nonprofit leaders, the EdTech Genome Project developed two tools (1) the EdTech Context Framework, comprehensive definitions for each variable, and (2) the EdTech Context Inventory, a set of 10 cohesive instruments for detecting and measuring these variables.
• With the release of these tools, educators and education stakeholders now have the shared language and measurement instruments needed to detect and document how and why the impact of edtech tools varies so widely from school to school." (p. 4)

If you want to evaluate technology and how effective it is being used in the classroom, consider answering the following questions that Steven Anderson (2015) provided:

1. Who is using the technology?
2. If you took the technology away, how different would the lesson be?  Anderson stated, "Ultimately technology should enable students to do something they couldn't do without it."
3. How much variety with the technology is there?  Are students using "different sites, apps, or programs?"
4. What opportunities do students have to collaborate with or through the technology?
5. What opportunities do students have to create new knowledge or products with the technology? (Anderson, 2015)

Steven Ross and Deborah Lowther (2009) identified three forms of technology applications that show promise for using "technology reflectively and scientifically to make teachers and curricula more effective."  These include "as a tutor, as a teaching aide, and as a learning tool" (p. 21).  The first two of those help teachers to address individual needs, and the latter can help learners acquire 21st century skills such as "searching the Internet, creating graphs and illustrations, and communicating through multimedia presentations" (p. 21).  As a tutor, computer assisted instruction can provide students with extra practice on key skills and content, provide remediation instruction, provide enrichment activities, and provide alternative ways to teach material for deeper learning.  As a teaching aide, tools such as whiteboards enable teachers to better orchestrate their lessons; clicker response systems enable timely feedback to questions that teachers pose (pp. 20-21).

Among key messages from a January 2012 joint position statement, Technology and Interactive Media as Tools in Early Childhood Programs Serving Children from Birth through Age 8, the National Association for the Education of Young Children and the Fred Rogers Center for Early Learning and Children’s Media at Saint Vincent College (2012) stated:

Effective uses of technology and media are active, hands-on, engaging, and empowering; give the child control; provide adaptive scaffolds to help children progress in skills development at their individual rates; and are used as one of many options to support children’s learning. Technology and interactive media should expand children’s access to new content and new skills. When truly integrated, uses of technology and media become routine and transparent—the child or the educator is focused on the activity or exploration itself and not on the technology.  (Key Messages Summary, p. 1)

While the above statement applies to technology use in early childhood programs, the position could be adopted for all learners.  I found in my own research that with any educational intervention, the effectiveness of technology depends upon the appropriate selection and implementation of that technology to meet teaching and learning goals.  Once selected, technology-use must be a regular, integral part of an instructional program and not viewed as an add-on in order to have a positive effect on achievement (Deubel, 2001). 

As an example, Project RED (2010), a national research initiative, contributed nine key technology implementation factors leading to academic success, specifically in reducing dropout rates, increasing graduation rates, reducing disciplinary actions and improving high-stakes test scores.  Its survey involving 997 schools with varying levels of technology integration and diverse student populations revealed the following in rank order:

1. Intervention classes: Technology is integrated into every class.
2. Principal leads change management and gives teachers time for both Professional Learning and Collaboration. (This occurs at least monthly, according to Leslie Wilson (2010), co-author of Project RED).
3. Games/Simulations and Social Media: Students use technology daily.
4. Core subjects: Technology is integrated into daily curriculum.
5. Online Assessments: Both formative and summative is done frequently. (Additionally, Wilson (2010) noted that online formative assessments are done at least weekly.)
6. Student-Computer Ratio: Fewer students per computer improves outcomes.
7. Virtual field trips: With more frequent use, virtual trips are more powerful. (According to Wilson (2010), the best schools are doing these at least monthly.)
8. Search engines: Students use daily.
9. Principal is trained via short courses in teacher buy-in, best practices and technology-transformed learning. (Project RED, 2010, slide 6; Wilson, 2010)

Owing to the personal, mobile, social, and networked nature of today's technology, opportunities to improve educational efficacy might lie in areas beyond personalized instruction systems, which have been the focus of much of the technology developed for schools, per Noel Enyedy (2014).  After more than 30 years of personalized instruction, research from large-scale studies and meta-analyses still reveals incremental change with "mixed results ranging from modest impacts to no impact" (p. 15).  Enyedy commented on a new metaphor of learning:

The type of computer technology that many believe will lead to transformational change will be technologies built around the process of learning and that attempt to enhance human-to-human interaction, not supplant it: technologies that spark conversations and inquiry; technologies that support these conversations with tools for visualization, simulation, analysis and communication; technologies that allow the students to create physical or computational objects; and technologies that allow students to share their ideas and solutions with their peers and larger social networks for feedback and refinement. (p. 16)

Technology Frameworks

Effective integration of technology into instruction is also linked to the framework used to guide the selection process.  The following frameworks might be considered.

The TPACK model addresses the complex relationship among technology, pedagogy, and content.  A change in any one of those three affects the other two, according to Punya Mishra and Matthew Koehler (2006).  As such they indicated, "there is no single technological solution that applies for every teacher, every course, or every view of teaching" (p. 1029).

Further, technological, pedagogical, and content knowledge (TPACK) is the basis of good teaching with technology.  In order to make a technology selection that has a chance at being effective, the teacher needs to consider those TPACK factors in relation to each other and should have acquired:

an understanding of the representation of concepts using technologies; pedagogical techniques that use technologies in constructive ways to teach content; knowledge of what makes concepts difficult or easy to learn and how technology can help redress some of the problems that students face; knowledge of students’ prior knowledge and theories of epistemology; and knowledge of how technologies can be used to build on existing knowledge and to develop new epistemologies or strengthen old ones. (Mishra & Koehler, 2006, p. 1029)

The SAMR (Substitution, Augmentation, Modification, Redefinition) model was developed by Dr. Ruben Puentedura and offers a method of seeing how computer technology might impact teaching and learning.  In SAMR and Bloom's Taxonomy: Assembling the Puzzle Puentedura (2014) elaborated on the nature of technology at each level, as follows:

• Substitution (S): Tech acts as a direct tool substitute, with no functional change;
• Augmentation (A): Tech acts as a direct tool substitute, with functional improvement;
• Modification (M): Tech allows for significant task redesign;
• Redefinition (R): Tech allows for the creation of new tasks, previously inconceivable.

The S and A levels are associated with the three lower levels in Bloom's Taxonomy, that is Remember, Understand, and Apply.  The M and R levels are associated with Bloom's three upper levels, that is Analyze, Evaluate, Create.  (Puentedura, 2014)

If you are not familiar with the SAMR model, the following might also help you:

• SAMR in 120 Seconds video posted on YouTube
• Introduction to the SAMR Model video posted on Common Sense Media.
• Using the SAMR model to frame how to teach with apps, a post with 21 ways to teach and learn with apps by the TeachThoughtStaff (updated 2018, September 24).
• Kathy Schrock has a collection of resources to learn more about SAMR and BLOOM'S.

The PICRAT model addresses two main questions for technology integration and includes a matrix, developed by Royce Kimmons, to help guide technology integration practices.

• What are students doing with technology?  Will they be passive (P), interacting(I), or creating (C)?
• What is the technology use's effect on practice?  Will it be a replacement (R), amplification (A), or transformation (T)?

To learn more, view PICRAT for Effective Technology Integration in Teaching on YouTube.

The Bottom Line of Tech Effectiveness

As a bottom line, Mark Schneiderman (2004), then Director of Education Policy at the Software & Information Industry Association (SIIA), confirmed "education technology is neither inherently effective nor inherently ineffective; instead, its degree of effectiveness depends upon the congruence among the goals of instruction, characteristics of the learners, design of the software, and educator training and decision-making, among other factors" (p. 30).  "Proper planning, teacher training, school leadership, technical support, configured hardware, network infrastructure and Internet access, pedagogy and instructional use, intensity of software use" (SIIA, 2009, p. 2) all play a role in an effective implementation.

Before purchasing any software/app, look for research that has been conducted into the effectiveness of the product on the achievement of learners.  Also consider contacting others who use the product to gain their perspectives on its use.

 

 

Back to top

 

What do you do, if you are not convinced you can integrate technology into your instruction?

Resistance and What to Do About It

According to David Roh (cited in O'Hanlon, 2009, p. 32), "There are hundreds of reasons why teachers don't want to use technology.  "While exact figures of how many teachers are not using technology in instruction are unknown, Charlene O'Hanlon (2009) pointed out that "anecdotal evidence from vendors and school districts alike indicates resistance to technology adoption is still a problem among a significant portion of the teacher population" (p. 32). They will resist if they are not shown the value that a technology will bring to the classroom, and if they are told they must use it and are given a deadline for doing so.  Their resistance might also stem from a fear of students knowing more than they do about a particular technology, which might happen if they lack a firm grasp of the technology.  So, before tackling resistance, it is important to understand the cause of resistance from the teacher's perspective.

In any approach to integrating technology in instruction, O'Hanlon (2009) suggested that teachers need to be able to learn a technology gradually, and be given time to learn it. Comprehensive training from vendors with follow-up professional development and support within the district will help resolve the resistance issue, as will a peer-to-peer mentoring program.  If all else fails, districts might even consider financial incentives for learning and adopting the technology.  For some, all it might take to convince a teacher to give the technology a try is for them to see how using the technology impacts students, and to witness the excitement of the early adopters.

In a nutshell, elementary principal Rob Furman (2013) would most likely call overcoming resistance to technology integration as "Do It on Their T.E.R.M.S." -- a great phrase he posed indicating what teachers need: Time, Encouragement, Resources, Modeling, and Shared success.

Before purchasing any software/app, involve teachers and potential users in the selection process.  As Katelyn Sweeney (2019) stated:

"Educators have insightful knowledge about their students’ needs and can be valuable assets in informing purchasing decisions.  If teachers are recognized as partners in the decision-making process, they are much more likely to support and implement the product in the classroom" (Item 5).

Activities to Try

The following activities should help convince you to give technology a try.

See the Technology Integration Matrix (TIM) developed for K-12 teachers in Florida.  The TIM has 25 cells created by associating five levels of technology integration (entry, adoption, adaptation, infusion, and transformation) and five characteristics of meaningful learning environments (active, collaborative, constructive, authentic, and goal directed).  Each cell includes a link to one or more videos that show technology integration in classrooms where only a few computers are available and/or classrooms where every student has access to a computer.  Descriptions of projects learners did and technology requirements are provided so that others might use the same project in their classrooms.  The matrix includes videos broken down by subject area, including math, science, social studies, and language arts, and by grade level.

Read Michael Orey's (2010) book, Emerging perspectives on learning, teaching, and technology.  This 349-page book is available online.  It delves into learning and cognitive theories, instructional theories and models, inquiry and direct instruction strategies, motivation, computer mediated instruction, learning communities, reciprocal teaching, technology tools for teaching and learning, and more.

Examine the professional development options at OTIS for educators (Online Technology and Instructional Sessions).  There is an extensive library of courses in over 40 categories to help you to integrate technology into instruction.  Among categories are Apple, Assessment, Blended Learning, Canvas, Digital Citizenship, Math, Project Based Learning, Social Emotional Learning, iPad, STEM, and more.  Some courses are free.  You'll find ISTE standards-aligned courses, closed captioned courses, courses leading to micro-credentialing, and more.  Many can be completed in under an hour.

 

Back to top

 

How can innovations, including technology, be sustained in schools?

 

There are five stages to go through for an innovation to be sustained per Richard Rush (2019), summarized as follows:

1. Knowledge: Individuals become aware of and interested in the innovation, perhaps from colleagues, peers, conferences, advertisements, blog posts, promotions from providers/advertisers, and so on.
2. Persuasion: Potential users or beneficiaries explore the capabilities and value of the innovation perhaps as a means for saving time, reducing costs, or improving performance.  Testimonials from users help promote positive perceptions.
3. Decision: Although a small group or one individual might make a final decision to adopt the innovation, collaboration helps others feel included in the decision.  It's important to bring all stake-holders into the decision-making process.
4. Implementation: Training helps with implementation.  Ease of use is essential.  Sharing testimonials from users on how the innovation has been successfully implemented into their activities should help motivate others to give it a try.
5. Confirmation: Per Rush (2019), "Success in the confirmation stage becomes evident when people are no longer creating work arounds and resorting to old processes."

Any innovation, including the implementation of educational technology, must become part of a school's culture to be sustained.  David Jakes (2006) stated:

1. The innovation must have multiple entry points for a spectrum of usership; each of these entry points must support effective use by teachers and students.
2. The teacher has become a confident, active, and visible user--use becomes seamless and transparent.
3. There must be a high degree of organizational readiness for the innovation.
4. The innovation must clearly address an instructional need, with benefits for both teacher and student.
5. The technology has been taken out of the technology, or innovation.
6. The innovation must add value to an instructional process.
7. There must be visible and tangible results indicating that the innovation improves student learning.  (Story 4: The Seven Factors of Stickiness section)

In support of and extending Jakes' (2006) view, Dian Schaffhauser (2009) noted six dimensions for scaling an innovation, which are in a framework developed by Chris Dede of Harvard University and Allyson Knox of Microsoft.  The traditional way to look at an innovation is spread.  However, a successful innovation program involves more than just adding users.  It involves depth, sustainability, shift, evolution, and emotion. Depth produces transformative change and leads to improved educational outcomes; Sustainability, as the name suggests, means that the changes in practice are maintained over time; Shift refers to the users of the innovation assuming ownership of it and spreading its impact to others; Evolution means that those who adopt the innovation make revisions to it as an ongoing process (p. 32).  Emotion comes into play when asking people to try something new.  While those who are struggling might be eager, those who believe themselves operating at high quality might not be so willing (p. 33).  The key to remember is that significant changes in classroom instruction brought about by a sustained innovation might not be realized for years.

In terms of sustaining technology in schools, a more concentrated effort is needed to use technology to customize learning.  According to the Digital Learning Council (2011), an initiative of the Foundation for Excellence in Education (ExcelinEd), "Today, less than 10 percent of students around the nation are experiencing the benefits of digital learning. States must advance bold reforms to make systemic changes in education to extend this option to all students" (p. 3).  Digital reform in education is needed to better meet needs of students who are already using technology out of school for such things as texting, gaming, posting on the internet, and exercising their own creativity using technology tools.  Ten elements for high quality digital learning are included within its Roadmap for Reform:

1. All students should be digital learners.
2. There should be no barriers to their access of digital content of high quality.
3. Students should be able to use technology to customize their learning.
4. Their progress should be based on demonstrated competency.
5. Content and courses should be of high quality.
6. Likewise, instruction should be of high quality.
7. They should have quality choices from multiple providers.
8. In terms of assessment and accountability, student learning should be the metric for evaluating the quality of content and instruction.
9. Funding should create incentives for performance, options and innovation.
10. Of course, the infrastructure should support digital learning. (10 Elements of High Quality Digital Learning section, p. 3).

There must also be a change in the teacher's paradigm for teaching and learning, according to Gary Shattuck (2013).  His Six Laws for Tech Adoption also relate to sustaining an innovation.  If an innovation is considered, educators must also deal with the laws of scarcity, change, beliefs, perception, diffusion, and leadership, which he proposed:

• Scarcity: If sufficient hardware, quality software, and bandwidth are not available for all users, an innovation could not be sustained.
• Change: Educators need to be willing to adapt to changes in society that impact "everything from economic issues to political, cultural, and societal issues."
• Beliefs: Teacher beliefs must be considered.  Shattuck noted "To fully embrace technology in education a teacher must adopt constructivist pedagogies. Without this adoption, technology in education will forever be used only at a superficial level."  By this Shattuck means that educators will best serve students when they believe in and use a predominantly student-centered classroom rather than a teacher-centered classroom.
• Perception: "Making decisions based upon your reality only can create problems. We must understand other people’s perceptions before moving forward."
• Diffusion: We need to know how to foster diffusion of an idea.
• Leadership: The school or district leader matters when considering the laws of scarcity, change, beliefs, perception, and diffusion. (Shattuck, 2013)

Next, what goes on in the classroom level is at the root of sustaining innovation.  Teachers need to be able to foster innovation and creativity in the classroom so that students become inspired.  Trevor Shaw (2015) outlined eight principles for the innovative classroom:

1. Give students a problem that is both interesting and authentic.
2. Give them the basics for their projects, but keep it short--chunked in about 10 minute blocks at the beginning of class.
3. Model research skills.
4. Scaffold complex skills.  Per Shaw (2015), "Tools like Makey Makey, Little Bits, Scratch, Tickle, and Tynker make it easier than ever for novice students to create authentic products that solve real problems" (p. 26).
5. Always check for understanding so that students don't become frustrated and discouraged.  Keep track of what students know and what they yet need to learn to successfully complete projects.
6. Favor found and recycled objects.  They don't always have just a single use.
7. Model mental inventory taking.  Students should be required to list the things they know about their projects and articulate what they need to understand better.
8. Above all, don't try to grade creativity and innovation.  "Innovation is not a standard that you can teach to directly and then test for.  Innovation is more like a habit of mind that is fostered through consistent attention to classroom culture and expectations" (Shaw, 2015).

Teachers might also need innovative ideas to try with their learners.  For this, Carl Hooker at HookED on Innovation provided 36 Weeks of Innovation for Your Classroom (2015) with innovations listed in order of difficulty ranging from using selfies to creating a start-up incubator.

As leadership matters per Shattuck (2013), districts should share their success stories as a means for sustaining a technology innovation and promoting technology integration, particularly for people who are still non-believers.  As suggested by The League of Innovative Schools (2014):

• "Ask community partners to help. The district tells the story first and then others repeat it. That is an effective way to spread positive messaging.
• Have students tell the story at the local level by creating videos and other multimedia content to show how they’re using technology in the classroom. Host a student film festival in your district on how students are using technology to learn.
• Find comparable districts that will help you tell your district’s story. Identify "doppelganger districts" - districts that look like your district - and share stories of their success until you have your own stories to tell.
• Bring in business leaders from the local community to be "principals for a day" in technology rich schools. This would cultivate partnerships, create a culture of understanding and collaboration, and enable business and education leaders to learn from each other. Invite the local media to cover the event." (p. 4)

In terms of infrastructure, the Consortium of School Networking (2019) found that a cloud-based infrastructure is among the top five tech enablers advancing teaching and learning innovation.  "A virtual infrastructure delivered or accessed via a network or the internet enables schools to move hardware and software services away from physical locations. Shifting to cloud services makes teaching and learning resources more readily available in any location—and it can reduce costs" (Tech Enablers Survey Results: Key Findings #3, 2019).  This brings up the issues of the amount of cloud-based storage needed, how data and access to it are secured in the cloud, and if institutions will use a commercial cloud-based service or have a private cloud infrastructure or a combination of both.

Education Week Research Center's 2019 national representative survey of preK-12 teachers (N=700) revealed "It is difficult for teachers to even consider ed-tech innovation if they lack such basics as reliable WiFi, adequate tech support, and sufficient numbers of devices" (2019, p. 6).  Fortunately, "A majority of teachers [said] they have access to all three at school" with respective responses being 72% for WiFi, 69% for tech support, and 56% indicating they had enough devices for every student to have his or her own (p. 6).  Unfortunately, access to technology outside of school hours poses a problem, as only 22% of respondents agreed that "they can safely assume that students can complete homework that requires access to technology outside of school hours" (p. 6).  Thus, results showed there is still room to grow for edtech innovation both in and out of school.  In spite of having available in-school technology, "What is less clear is the degree to which teachers have used these new tools [high-speed WiFi, cloud-based services, low-cost devices] in innovative manners, rather than merely continuing their former practices on new and different platforms" (p. 5).

Digital Promise with input from the Verizon Innovate Learning Schools partner districts developed a Technology Sustainability Toolkit, which should help districts in allocating available funding.  The toolkit includes additional information and resources within the following steps:

• Evaluate available funding
• Consider your district's overarching goals
• Identify your district's current technology reality
• Identify district technology growth plans
• Collaborate to support learning
• Determine the total cost of ownership of your solutions
• Review products
• Build your lifecycle plan
• Build systems of support around implementation
• Share the value of investment
• Commit to sustaining the plan

Finally, Dr. Matthew X. Joseph and Dr. Chris Jones (2023) also emphasized the role of leadership in supporting an innovation in schools.  School leaders should stay informed by keeping "up-to-date with the latest educational research, trends, and best practices to identify opportunities for innovation" and "hear multiple perspectives on every aspect of the educational experience."  They should provide "relevant, ongoing, and job-embedded professional development" that focuses on "innovative teaching methods, technology integration, and new educational trends."  School leaders should promote blended learning, as "Encouraging a blend of traditional classroom instruction and online learning provides flexibility that allows students to learn at their own pace and style while providing teachers with data to personalize instruction."  There should be classrooms with flexible learning spaces, as these are "adaptable to different teaching and learning styles" and "can foster creativity and collaboration." (p. 13)

 

Back to top | Part 1: Technology Integration: Essential Questions: Page 1  |  2  |  3

 

References

 

Back to top

See other Technology Integration pages:

Part 1: Technology Integration: Essential Questions: Page 1  |  2  |  3

Part 2: Technology Integration Resources  |  Part 3: Web Page Design  |  Part 4: Multimedia in Projects

 

 

 

 

Comments?  Are you finding resources at CT4ME of value?

Send us your favorite math resources and tips for success.

Contact Dr. Patricia Deubel: pdeubel@ct4me.net

To cite this page, use the following format:

Deubel, P. (year from last revised). Technology integration: Essential questions. Computing Technology for Math Excellence. Retrieved Month, Day, Year, from https://www.ct4me.net/technology_integr_2.htm

CT4ME Author: Dr. Patricia Deubel

CT4ME Privacy Policy

Copyright © 2001-2024 [CT4ME.net].  All rights reserved.