Augmented Reality Weather App

What are you doing when you want to know the current weather and maybe with a slight prediction how it will change?

Obviously, you pull out your smartphone and look it up, but what if you don't have it on you, you look up in the sky, the most natural way for us to see how the weather is going, humans are doing this since the dawn of time.

So why not create a fusion about these two ways to gain information about Weather?
I asked this at denkwerk and so the Tenkigo project was kickstarted.
Tenkigo is Augmented Reality Weather Forecast App which combines the natural way of looking up in the sky with the information dense way a app provides.
Why Tenkigo? Tenki is japanese for weather and "go" is for going outside.

Imagine a future were Augmented Reality Glasses are common sight. Tenkigo shows what this could mean for rich information like weather.

Technology and Creativity combined

Niantic ARDK

The key technology is Niantics ARDK under the Niantic Lightship umbrella, its a Augmented Reality Toolkit for Unity developed by the same people that brought us Pokèmon Go.
The feature that made Tenkigo possible is called Semantic Segmentation, it splits up the camera feed into different layers, like ground, trees, buildings and the most important one for us, sky.

Weather API

For the weather data, I looked at different weather APIs and after some research I decided to use the OpenWeatherMap API, it has a generous free tier and the data you can get with only one API call is quite extensive. Caching the results for all 48 hours of forecast results in even less calls per app instance. So this means we could have 1000 app openings per hour and not pay a single cent. Amazing!

Optimization

One of the biggest challenges was to optimize the recording and encoding to a GIF. 
The whole recording process had to be multithreaded, capture the raw GPU frames, write them into internal system storage so the main thread would not be slowed down.

The first version of this process saved .bmp files on wrote them temporarily into the system storage, Bitmap Files are fast to encode and relatively fast to decode but still resulted in a slowdown of the IO process, so I decided to use RAW GPU frames instead.

After recording the gif encoder process reads these raw frames and asynchronously encodes them to a GIF image.
The GPU frames are big uncompressed files but are extremely fast to write and to read because there is no encoding and decoding step which slows down the IO process immensely.
The problem was that Unitys default garbage collection behaviour stored them way to long in memory therefore when a second recording was started the application crashed.
Manual garbage collection at defined times overcome this challenge and cleared enough memory so the application can continue indefinitely.

Have a look

We made a small teaser for Tenkigo.