Home Teach and assess Classroom resources Lesson ideas Home automation: General purpose programming
home-automation-programming_thumbnail.jpg

About this lesson

Investigate home automation systems, including those powered by artificial intelligence (AI) with speech recognition capability. These suggested activities for year levels 7-8 are designed for students using general purpose programming languages JavaScript and Python, with similar content to the visual coding lesson Home automation programming for year levels 5-6.

Year band: 7-8

Curriculum Links Assessment

Learning sequence

Download Lesson

Before you begin

The use of speech recognition with Python and JavaScript brings specific hardware and software requirements. See the instructions below. (Note, the same activities are provided for both languages).

Before using any new technology tools undertake a risk assessment using the eSafety Commissioner’s New Technology: Risk assessment. 

Javascript

Students will use online environment Pencil Code for speech recognition.

  • Computer / laptop with microphone required.
  • Chrome browser required.
  • Pencil Code launches with a less common language CoffeeScript by default. Switch to JavaScript using the cog button above the code area Image of a cog. Blocks for JavaScript also become available once all CoffeeScript code is removed.

Python

Speech recognition is not currently available in an online environment for Python.

  • Computer / laptop required.
  • Microphone use is not available. Students must pre-record audio clips of their vocal commands (eg. using a tool like Audacity) and save these in the folder with their Python programs.
  • Software installation required. We recommend Thonny, a simple IDE that comes with Python built in and makes additional modules easy to install.
    • Download and run the Thonny installer from thonny.org
    • Start Thonny.
    • Click Tools → Manage packages...
    • In the search bar at top, type speechrecognition, then click Find package from PyPI.
    • Click Install to add the Speech Recognition module to Thonny.
    • In the search bar at top, type playsound, then click Find package from PyPI.
    • Click Install to add the playsound module to Thonny.
    • Click Close to return to the Thonny main screen.
  • Optional: To investigate the use of microphone in Python, try this webpage. The compatibility of audio modules for microphone use varies by Python version and microphone use requires installation of additional software on Windows and other platforms.

Test your setup

Javascript

To test speech recognition at Pencil Code:

  1. Visit dthesa.pencilcode.net/edit/speechTest to load a simple JavaScript program.
  2. Press play to run the program. It should record something you say, then say it back to you.

If the program does not work:

  • Check the hardware and software requirements under Before you begin.
  • Adjust the microphone or speaker volume on your computer.
  • Disable any other software that might be using your microphone, eg. a web chat.
Privacy warning!

Avoid students recording their own voice in the pencil code environment. Generally an AI speech recognition system records the voice to then compare to its database of known speech patterns and data. The recordings would be stored on a server. This digital file may not be deleted.

To mitigate risk of a student’s voice being saved, use a device that has text to speech functionality. For example open an app on your tablet device or in your browser such as https://www.naturalreaders.com/online/. Type text in and play speech. The pencil code should operate based on this speech input.

 

Python

To test speech recognition in Thonny:

  1. Ensure you have installed Thonny and additional modules as per the instructions under Before you begin.
  2. Download 'testing123.wav', an audio clip with voice saying "Testing 123".
  3. Download 'speech_test.py', a python program, and place it in the same folder as 'testing123.wav'.
  4. Start Thonny and open the program 'speech_test.py'.
  5. Run the program.

If the program does not work:

  • Check the hardware and software requirements under Before you begin.
  • The folder with the two files may be buried deep in a folder hierarchy with long names. Try moving the files to a simpler folder before opening again in Thonny.
  • Adjust the speaker volume on your computer.

Suggested steps

Unplugged activity

Discuss home automation. Ask, ‘How might voice commands be used in home automation?’ Consider aspects of home automation such as:

  • security and access
  • climate control
  • lighting
  • entertainment.

Discuss the role of AI in performing the automation. This may prompt students to mention the use of personal assistants.

Ask students to consider how home automation can or might assist those living with disability.

With new technologies there are often risks to consider. Ask what could go wrong. Discuss risks such as privacy breaches from a system being hacked or times when the system might be down due to a power outage (including one caused by a natural disaster such as fire or flood).

Plugged activity

Step 1: Activating and deactivating a light

Use these suggested activities to explore implementing a digital solution that demonstrates how to control appliances, and to further investigate home automation.

Use binary numbers 0 and 1 to switch a light on and off (skill level: easy)

Use JavaScript or Python (without any form of AI) to create a program that switches appliances on and off using an input: 0 or 1. This task demonstrates the use of binary digits to change state; for example, ‘On to off’ or ‘Off to on’. As this program does not use any form of AI, use it to demonstrate how conventional coding is used to hard-code a particular action.

Depending on students’ familiarity with and understanding of binary numbers, review the use of 0 and 1 and explain that binary means ‘two states’. The two states can be represented in different ways, such as by the numbers 1 and 0, or by text (‘true’ and ‘false’, or ‘on’ and ‘off’). The main point here is that a binary device can be in just one of two possible states. A binary bit is a single on/off value.

Abstraction and representation Abstraction and representation Click to learn more

Follow the steps below to implement the program at Pencil Code. (Be sure to read 'Before you begin' and 'Test your setup' first.)

Click here for the completed Step 1 program.

  1. Start a new JavaScript program at Pencil Code. Use this shortcut to an empty JavaScript program, or follow these steps:
    1. Go to pencilcode.net.
    2. Click Let's Play!
    3. Click the cog button image of cog above the code area and select JavaScript.
    4. Delete any existing CoffeeScript code.
  2. Write a program to show a light going on and off when the user types "0" / "on" or "1" / "off". This can be done with two simple functions activated by keydown events, which change the image for a sprite. 
        // Create and position the sprite.
var light = new Sprite();
light.jumpto(-150, 0);

// Show the light on.
function showLightOn() {
    light.wear('https://bit.ly/light-on-image');
}

// Show the light off.
function showLightOff() {
    light.wear('https://bit.ly/light-off-image');
}

// Main program.
light.wear('https://bit.ly/light-off-image');
write('Press 1 to turn the light on, or 0 to turn it off.');
keydown('1', showLightOn);
keydown('0', showLightOff);

Follow the steps below to implement the program in Thonny. (Be sure to read 'Before you begin' and 'Test your setup' first.)

  1. Start a new Python program in Thonny. Save the file with a relevant file name, for example light_switch_text_input.
  2. Write a program to demonstrate a light going on and off when the user types "0" / "on" or "1" / "off". This can be done with a simple loop that runs forever. (Use Thonny's STOP button to end the program.)
    print('Type 1 to turn the light on or 0 to turn it off.')
while True:
  command = input()
  if (command == '1'):
    print('The light is ON.')
  if (command == '0'):
    print('The light is OFF.')
  • Try adding a second appliance with a different input command to turn it on or off.
Step 2: Adding speech recognition

Present the challenge of creating a program that recognises the students’ voice commands to perform an action such as turning home appliances on or off.

Discuss the types of voice commands that might typically be used, for example:

  • Turn on/turn off
  • Lights on/Lights off
  • Turn fan on/Turn fan off.

Emphasise short commands that are easy to understand.

Follow the steps below to implement the program at Pencil Code. (Be sure to read Before you begin and Test your setup first.)

Click here for the completed Step 2 program.

  1. Begin with the completed Step 1 JavaScript program at Pencil Code.
  2. Modify the program so that:
    • a single function is triggered by listen,
    • the function checks the recognised speech text for keywords,
    • it changes the sprite based on whether keyword 'on' or 'off' is found,
    • finally, it sets up the listen trigger again.
    // Create and position the sprite.
var light = new Sprite();
light.jumpto(-150, 0);

// Change the light based on speech text.
function changeLight(speechText) {
  if (speechText.includes('on')) {
    light.wear('https://bit.ly/light-on-image');
  }
  if (speechText.includes('off')) {
    light.wear('https://bit.ly/light-off-image');
  }
  listen(changeLight);
}

// Main program.
light.wear(''https://bit.ly/light-off-image');
write('Speak to turn the light on and off.');
listen(changeLight);

Follow the steps below to implement the program in Thonny. (Be sure to read 'Before you begin' and 'Test your setup' first.)

Click here for the completed Step 2 program.

  1. Have students create audio files with vocal commands like 'Light switch on' and 'Light switch off'. These should be saved as WAV files in the same folder as your Python programs. (Or download premade audio files: On, Off).
  2. In Thonny, start with the completed Step 1 Python program and save with a fresh filename.
  3. Modify the program to include the speech recognition module. We'll still have the user type 1 or 0, but now it will activate a function to load and play the relevant audio clip then try to recognise the voice command. 
    import speech_recognition
import playsound

# Play audio clip and attempt to recognise speech.
def recogniseCommand(filename):
  print('Now testing', filename)
  playsound.playsound(filename)

  recognizer = speech_recognition.Recognizer()
  source = speech_recognition.AudioFile(filename)
    audio = recognizer.listen(source)    
  text = recognizer.recognize_google(audio)
  
  print('I heard:', text)    
  if 'on' in text:
    print('The light is ON.')
  elif 'off' in text:
    print('The light is OFF.')
  else:
    print('Voice command unrecognised.')


  # Main program.
  print('Type 1 to turn the light on or 0 to turn it off.')
  while True:
    command = input()
    if (command == '1'):
    recogniseCommand ('lightSwitchOn.wav')  # Use your audio clip filename here.
    if (command == '0'):
      recogniseCommand ('lightSwitchOff.wav')  # Use your audio clip filename here.
  • Add an activation phrase like "OK, House" or "Hey, House". Commands to turn the light on or off will be ignored until this phrase is first recognised.
    • For JavaScript:
      • Click here for a completed solution to this tinker idea.
    • For Python:
Challenge: Multiple appliances

Create a program that recognises the students’ voice commands to turn multiple home appliances on or off.

Discuss the types of voice commands that might typically be used, for example:

  • Turn on/turn off
  • Lights on/Lights off
  • Turn radio on/Turn radio off.

Emphasise short commands that are easy to understand, or keywords to scan for.

Click here for a completed sample solution.

Download premade .png files: "Radio on", "Radio off".

Download premade audio files: "Radio on", "Radio off".

Click here for a completed sample solution.

Discussion

Share what you have learned about AI and how ‘smart’ a computer can be. Think about the program you created:

  • What were some of the challenges?
  • Where the program used AI, what things affected how well it worked?
  • How might you improve the program if you did the task again?

Why is this relevant?

Algorithms and programming are essential to developing machines powered by artificial intelligence (AI). AI is the ability of machines to mimic human capabilities in a way that we would consider 'smart'.

In conventional programming the computer is provided with a set of instructions for a defined set of scenarios. In the binary program, the students hard-coded the program with specific inputs of 0 or 1 to turn the appliance off/on. The next program included speech recognition.

  • In the JavaScript program the user's speech was detected via a microphone (input) and this command was recognised using a form of AI, Natural Language Processing (NPL).
  • In the Python program the user's speech was pre-recorded as a WAV file, which provided the input. The program recognised the command using a form of AI, Natural Language Processing (NPL).

Machine learning (ML) is an application of AI. With machine learning, we give the machine lots of examples of data, demonstrating what we would like it to do so that it can figure out how to achieve a goal on its own. The machine learns and adapts its strategy to achieve this goal. In our example there are many variations on commands to turn appliances on and off. The use of short clear and precise commands increases the accuracy of the AI speech recognition.

This lesson focuses on:

  • binary numbers (1, 0)
  • speech recognition

Resources

  • Online JavaScript and CoffeeScript programming environment Pencil Code
  • Offline Python programming environment Thonny