22 Nov WebRTC Mobile at KrankyGeek
Kranky Geek 2016 in San Francisco was the place to be if you’re interested in WebRTC mobile development. I’ve been at many different WebRTC events over the last few years since starting RealTimeWeekly.co and WebRTC.ventures, but this was the most information-dense WebRTC event I’ve been to!
The event was held at Google’s San Francisco office again, and was packed with over 6 hours of content, all of which you can see you on YouTube.
Below I will summarize some of the key takeaways I had from each of the session speakers. If you’re interested in the details of WebRTC mobile development, you should definitely watch the videos though because there is a lot more information and useful advice than I will cover in these notes!
First let’s start with an introduction to mobile development for WebRTC from Tsahi Levent-Levi, one of the event organizers.
WebRTC on Mobile: What are your choices?
Tsahi Levent-Levi, BlogGeek.me
From there, the conference went into deeper dives on specific WebRTC mobile topics, which I summarize below.
Android Development with WebRTC
César Guirao, Tokbox
There are 3 main ways you can build an Android WebRTC app:
1) android webview – lollipop (5.0) or newer
2) Native WebRTC Java API – more complex API, must compile WebRTC yourself
3) C++ API – max portability since can run C++ anywhere, but complex to work with and hard to maintain
You can see the code César showed in this repo: https://github.com/Nitrillo/krankygeek
To reduce the size of the Android binary you create, use the “split mechanism” option to create different APK’s for each architecture when compiling. Because of the proliferation of Android devices, César noted that you really need a test lab with physical devices for android testing of WebRTC.
iOS Development with WebRTC
Arik Halperin, Greenfield Tech
You can use the AppRTC pod to build WebRTC for your app, or build your own pod or SDK.
AppRTC is open sourced by Google on github here:
It uses the pristine.io implementation of AppRTC, which is not always up to date with latest WebRTC standards.
Push Kit is also an option to use in your app – This way your app is not always running with an open socket since that will drain the battery. Instead uses VOIP Push for maximum priority on the OS to receive your calls.
Also you can use the iOS Call Kit – This makes your incoming call appear like any other native call.
Video communications on iOS
Chris Eagleston, Twilio
Twilio builds on the C++ API’s in iOS. Echo cancellation and gain control is done by hardware.
In addition to sharing some lessons learned with CallKit, Chris also talked about how to do screensharing in iOS. One important note about screensharing in iOS is that your app cannot share the screen if it’s running in the background. This effectively limits your ability to share the screen outside of your own application.
The complete CallKit example from Chris, and coming soon the Screen Recorder Demo
Adopting WebRTC’s C++ for cross platform development
Andrew MacDonald, Slack
Andrew gave a very interesting presentation about how Slack uses WebRTC in their application, and began by showing how the Slack architecture works.
But if you already have Java or Objective C apps then Andrew suggested you consider using Djinni and C++. Djinni generated 5k lines of code for them, and it was pretty good generation. This allowed for mostly cross platform development.
Write-once, run WebRTC anywhere with React Native
Emil Ivov and Lyubomir Marinov, Atlassian
Emil and Lyubomir both highly recommended the use of React Native for WebRTC, based on their experience using it at Atlassian for the Jitsi Meet application. Their talked included many lessons from their experience and is required viewing if you are thinking of building with React Native.
Among other things, they noted there are no simple return values and there is kind of callback hell again when building with React Native. Because of this, they noted that garbage collection is very important, and the MediaStream objects had to be released explicitly to prevent memory leaks.
If you like, then using JitsiMeet means you can build around what they’ve already done since it’s open source.
They were able to achieve 90% shared code between iOS and Android with React Native, which is pretty strong for any cross platform language. Although it’s currently still a work in progress, they are hoping they can achieve a similar share % on desktop too.
Improving mobile WebRTC video using Scalable Video Coding (SVC)
Gene Gershanok, Vidyo
In Scalable Video Coding, applications takes a single media stream can individual frames can be pulled out into multiple layers, which makes it easier for the router to determine which frames to drop under poor conditions. Gene gave a very interesting presentation on how this works, and showed how it adds better error resiliency. If packets need to retransmit then SVC requires less packets are retransmited. Vidyo has been working closely with Google on VP9 for WebRTC to include SVC.
Mobile troubleshooting & reverse engineering (of Google Duo)
Philip Hancke, Appear.in
Philip showed his usual wireshark magic to analyze how a closed WebRTC application works and share those learnings with the WebRTC community.
A few cool features of Duo that he focused on in the presentation:
1) Connection setup – Google Duo goes to TURN first to speed up the connection
2) Knock-knock – Requires high quality video instantly in 1st 2 seconds. Duo achieves this by measuring bandwidth early to know what to send
3) 4G-Wifi switch – Duo immediately goes back to the TURN server on same connection as previously used when a connection drops. This solves the “walk out the door” problem. It doesn’t use ICE restart, just keeps the connection open with the TURN server over mobile as a fallback to the wifi connection.
Philip had many good things to say about Duo and noted it is a good benchmark to test the quality of your app against.
Speaking of his company, Appear.in, he noted that their service runs TURN servers with geolocation and Route 53, which is a relatively easy and standard way to reduce latency by using the TURN server closest to the call participants.
WebRTC: News, Stats, and Audio Processing Internals
Niklas Blum, Justin Uberti & Per Ahgren from Google
The Google team started with a few interesting statistics that show how prevalent WebRTC has become:
>1 billion combined audio/video minutes each week in Chrome only
>1 petabyte per week passes through the datachannel
Tsahi Levent-Levi has tracked >1200 webrtc based companies and products
The Google team has been working on performance improvements as always, and one example is that Chrome has been able to reduce average ramp up time from 2 seconds down to 650ms for a 1Mbps connection. They are also working to bring VP9 to mobile devices, which will allow for high video quality with less bandwidth required.
Chrome version M54 added tab sharing, in addition to window and desktop sharing.
They are also working on traversing proxies that require authentication. This will help improve the availability of WebRTC in corporate networks.
They also presented interesting details on the audio processing pipeline, how different modules for echo cancellation and gain control work together. If this is of interest to you, definitely watch the video above for a full description.
Building a secure IoT device with WebRTC
Tim Panton, CTO of |pipe|
The final presentation of the day came from Tim Panton, who has done dangerously live coding demonstrations before at Kranky Geek and always presents an interesting application of WebRTC.
In this session, he showed how to build a webcam that cannot be accessed by an unauthenticated device. The device must be authenticated via a QR code, and his implementation essentially means a mobile phone or laptop can only exchange keys with the webcam if they are physically located by each other so the QR code can be scanned from an LCD on top of the Raspberry Pi running the webcam. If my summary doesn’t make sense, just watch the video and Tim will show you how it works.
This means a secure and private webcam with no open ports, minimal central service, and no passwords.
You could use this QR concept to connect to any IoT device securely, not just a webcam. In his demo, Tim used it to control a public piece of art in Reno, which allowed him to remotely change the color of lights on the display.
Check out the full demo in his video above.
As always, KrankyGeek was a great event, and this one was particularly useful for those interested in mobile development with WebRTC – which is probably most of you! For more information about future events, and videos of all their past speakers, check out KrankyGeek.com.