### Introduction to Using Cell Phone Sensors for 3D transformations

It’s a well-known fact that our mobile devices come equipped with all kinds of sensors used for various features that we tend to take for granted. For example, a phone knows if it’s held in portrait or landscape orientation, it knows if you’re walking in a certain direction, and it can even detect if you’ve left it on a table screen-up or screen-down. All of this information is calculated internally using various electronic sensors. But, as game devs can we make use of these sensors for our video games? The answer is yes, and it’s relatively straightforward.

In this Godot tutorial, you will learn how to use mobile phone sensors to control the transformations of objects in the Godot Engine.

You can get a demo project with two scenes to test the content of this tutorial on the Ramatak GitHub!

### Scope of this Tutorial

This Godot tutorial will cover the following topics:

- Understanding the concepts of pitch, roll, and yaw in 3D space.
- Using the gyroscope to set the pitch, roll, and yaw.
- Using the accelerometer and magnetometer to set the pitch, roll, and yaw.
- Combining the gyrroscope, accelerometer, and magnetometer for better control.

Throughout the post, we will use these values to control the camera node of the scene.

### Prerequisites for this Tutorial

To follow this tutorial you will need:

- Some knowledge of the Godot Engine.
- A mobile phone with
**gyroscope**,**magnetometer**and**accelerometer**. - Basic math skills (trigonometry mainly).
- Basic physics knowledge.
- Ramatak Mobile Studio (automatically setup for exporting to Android) or Godot 3.5.1 set up to export to Android.

### Setting up the Project

The project required for this tutorial is simple. Just use a *Spatial* node as the root of your scene, and add a *Camera* node to it. You should also add some meshes to have as a reference for when we start moving the camera. Your base project should look something like this:

### A First Approach: The **Gyroscope**

We’re going to use sensors to orient the camera in our game. To do this, we need to understand some basic definitions of camera rotation. With the axis convention used in the Godot Engine, we can define the following:

**Pitch**: rotation around the**x-axis**.**Roll**: rotation around the**z-axis**.**Yaw**: rotation around the**y_axis**.

Our goal is to determine these three parameters using our cellphone sensors.

Let’s start by looking at the **gyroscope**. A **gyroscope** gives us the **angular velocity** in radians per second (`rad/s`

). This means that the sensor tells us how fast the device is rotating on each axis. As we know from basic physics, multiplying *velocity* by *elapsed time* gives us the *distance* traveled in that time. The same principle applies to **angular velocity**: multiplying it by elapsed time gives us the change in angle in that time. Godot makes it easy to get this value with its `Input`

system. So let’s use this to get the **pitch** for our camera.

Add a script to the camera with the following code:

extends Camera var pitch: float = 0.0 func _physics_process(delta): var gyroscope: Vector3 = Input.get_gyroscope() pitch += gyroscope.z * delta rotation.z = pitch

Run your project now on your device, and move it around. You will notice that the camera moves accordingly!

So lets complete the previous script for **roll** and **yaw**:

extends Camera var pitch: float = 0.0 var roll: float = 0.0 var yaw: float = 0.0 func _physics_process(delta): var gyroscope: Vector3 = Input.get_gyroscope() pitch += gyroscope.x * delta roll += gyroscope.z * delta yaw += gyroscope.y * delta rotation = Vector3(pitch, yaw, roll)

Try it again and see that now you can freely rotate your camera to inspect the scene! You can now control the orientation of your camera using your mobile phone. Great! But we’re not done yet.

If you played around enough with the previous code, you will soon note that if the roll is

, **π**/2**yaw** and **pitch** stop working as desired. This is because, in the **object coordinates**, if `roll = `

, the **π**/2**yaw** should be considered around the **x-axis** since this is the axis parallel to the earth now. Similarly, the **pitch** should now be considered around the **y-axis**. This can be fixed by applying the **roll** also to the **gyroscope** data:

var gyroscope: Vector3 = Input.get_gyroscope().rotated(-Vector3.FORWARD, roll)

*Note: Remember that in Godot Vector3.FORWARD is a vector pointing in the opposite direction of the z axix.*

So, are we done now? Unfortunately, no…

Gyroscopes are cool sensors, but they have a problem: they tend to drift. If you move your camera around a lot and then put your phone in the same orientation it had when the game started, likely, you won’t get the same view as when the game launched.

### Using the **Accelerometer** for **Pitch** and **Roll**

The **accelerometer** is a sensor that, measures *acceleration* in units of `m/s²`

. In a rest position, **accelerometers** will measure the `9.81 m/s²`

in the direction to the center of the earth caused by gravity. This makes them useful to detect the orientation of a device relative to the earth.

Godot provides two measurements with the **accelerometer**:

`Input.get_accelerometer()`

: gives you the sensor values in each axis.`Input.get_gravity()`

: gives you the values but only takes into account gravity.

The latter is better for measuring **pitch** and **roll**.

We can use the *gravity* value to get the **pitch** and the **roll**, but we won’t be able to get the **yaw** as the gravity resultant is always parallel to the **y-axis**.

If you don’t care much about the math, just use this code and skip the next section:

func _physics_process(delta): var gravity: Vector3 = Input.get_gravity() roll = atan2(-gravity.x, -gravity.y) gravity = gravity.rotated(-Vector3.FORWARD, rotation.z) pitch = atan2(gravity.z, -gravity.y) rotation = Vector3(pitch, yaw, roll)

#### Math Justification

Trying to not go very deep with math, this picture is provided to help visualize the situation, which assumes that the **roll** is equal to 0 (the **x-axis** is parallel to the ground) and only considers the **z** and **y** axes.

The picture shows the local axes of the device when some **pitch** is applied. By definition, the **pitch** is the angle between the horizontal and the **z-axis** (i.e. the rotation around the **x-axis**). The magnitudes and represent the gravity measurements in those axes, as read by the **accelerometer**. The resultant gravity should always be a vector pointing downwards with a magnitude of approximately

.**9.81 m/s²**

By inverting** g**, we can get

_{y}

**and using trigonometry, it can be seen that the angle between**

`g`

‘,**-g**_{y}

and **is equal to the**

`g`

‘**pitch**. This angle can be calculated using the formula

`atan2(gravity.z, -gravity.y)`

.To account for cases when **roll** is not equal to 0, the gravity vector needs to be rotated by the **roll** value, which can be done using the code `gravity = gravity.rotated(-Vector3.FORWARD, rotation.z)`

.

The formula for the **roll** value can be obtained through a similar analysis, and is given by `roll = atan2(-gravity.x, -gravity.y)`

.

### Using the Magnetometer for **Yaw**

To determine the **yaw**, we use the **magnetometer**. This sensor measures the *magnetic fields* and is capable of detecting the direction of the north pole due to the earth’s magnetic field. By considering it as an accelerometer that points towards the north, we can calculate the **yaw** value in a similar way than we did for **roll** and **pitch**:

yaw = atan2(-magnet.x, magnet.z)

However, there are two factors to consider when calculating **yaw**. First, we need to rotate the readings from the magnetometer to take the **roll** and **pitch** into account.

var magnet: Vector3 = Input.get_magnetometer().rotated(-Vector3.FORWARD, rotation.z).rotated(Vector3.RIGHT, rotation.x) yaw = atan2(-magnet.x, magnet.z)

Secondly, to align the **yaw** value with the world coordinates in our game, we need to calculate an initial **yaw** value and subtract it from all subsequent calculations. The resulting script is:

extends Camera var pitch: float = 0.0 var roll: float = 0.0 var yaw: float = 0.0 var initial_yaw : float = 0.0 func _ready(): yield(get_tree(),"idle_frame") var magnet: Vector3 = Input.get_magnetometer() initial_yaw = atan2(-magnet.x, magnet.z) func _physics_process(delta): var magnet: Vector3 = Input.get_magnetometer().rotated(-Vector3.FORWARD, rotation.z).rotated(Vector3.RIGHT, rotation.x) var gravity: Vector3 = Input.get_gravity() roll = atan2(-gravity.x, -gravity.y) gravity = gravity.rotated(-Vector3.FORWARD, rotation.z) pitch = atan2(gravity.z, -gravity.y) yaw = atan2(-magnet.x, magnet.z) rotation = Vector3(pitch, yaw - initial_yaw, roll)

*Note: you need to wait for a frame before getting the magnetometer reading in _ready(), otherwise it will return Vector3.ZERO.*

We achieved our camera movement using another approach, but this method is not perfect either. As previously mentioned, **gyroscopes** have *drift* issues, but **magnetometers** and **accelerometers** aren’t a silver bullet either: they have *noise*. If your sensors are cheap as mine, you will notice the camera orientation is not smooth at all, particularly in regards to **yaw**.

**Gyroscopes** have little *noise* but a lot of *drift*, and **magnetometers** and **accelerometers** have little *drift* but a lot of *noise*. Fortunately, mathematical techniques allow us to combine the benefits of both sensors, providing a better solution!

### Improvement Using the Three Sensors

There are several methods for combining the readings from the three sensors to get a more accurate estimation of orientation values. However, for our purposes, the simplest solution will suffice.

We can get **roll**, **pitch**, and **yaw** from both the **gyroscope** and the **accelerometer**/**magnetometer**. By taking a *weighted average* of these values, we can mitigate the *drift* from one measure and the *noise* from the other. For example, to calculate the **roll** value, we can use the following code:

`roll = k (roll + gyroscope.z * delta) + (1-k) * roll_acc`

Here, `k`

is a constant between 0 and 1, and `roll_acc`

is the **roll** value calculated with the accelerometer. A value of `k`

close to 1 will result in the **gyroscope** being primarily used to calculate the **roll** while in motion, with the **accelerometer** “correcting” the estimation over time, especially while stationary. This is known as a *complementary filter*.

There is one final challenge: the **roll** calculated with a **gyroscope** can increase without bonds, but the one calculated with the **accelerometer** is limited to the range ** [-π, π]**. This can result in “jumps” in the values when the gyroscope’s

**roll**exceeds

**π**, as the

**accelerometer**will then report a value near –

**π**.

Godot comes to the rescue again! The *complementary filter* is essentially a linear interpolation between the two values, and Godot has a function specifically designed to interpolate between angles while wrapping them correctly. This function is `lerp_angle()`

and in this case, we can use it like this:

roll = lerp_angle(roll_acc, roll + gyroscope.z * delta, k)

With this in mind, the final code results in:

extends Camera var pitch: float = 0.0 var roll: float = 0.0 var yaw: float = 0.0 var initial_yaw : float = 0.0 var k : float = 0.98 func _ready(): yield(get_tree(),"idle_frame") var magnet: Vector3 = Input.get_magnetometer() print(magnet) initial_yaw = atan2(-magnet.x, magnet.z) func _physics_process(delta): var magnet: Vector3 = Input.get_magnetometer().rotated(-Vector3.FORWARD, rotation.z).rotated(Vector3.RIGHT, rotation.x) var gravity: Vector3 = Input.get_gravity() var roll_acc = atan2(-gravity.x, -gravity.y) gravity = gravity.rotated(-Vector3.FORWARD, rotation.z) var pitch_acc = atan2(gravity.z, -gravity.y) var yaw_magnet = atan2(-magnet.x, magnet.z) var gyroscope: Vector3 = Input.get_gyroscope().rotated(-Vector3.FORWARD, roll) pitch = lerp_angle(pitch_acc, pitch + gyroscope.x * delta, k) yaw = lerp_angle(yaw_magnet, yaw + gyroscope.y * delta, k) roll = lerp_angle(roll_acc, roll + gyroscope.z * delta, k) rotation = Vector3(pitch, yaw - initial_yaw, roll)

And that’s it! With this method, you can now control the `Camera`

or any other node that inherits from `Spatial`

using the sensors in your phone. The possibilities for creative game design with the Godot Engine are endless! Have fun experimenting with this technique.