Month: September 2021

Kurento Media Server: Everything You Need To Know In 2023

Post author By Grigoriy S
Post date 22.09.2021
6 Comments on Kurento Media Server: Everything You Need To Know In 2023

A short review on Kurento in simple language for business people and those who’re not into all that techy stuff.

Imagine that a programmer approached you and said that he needs a media server for development, and he recommends that you use Kurento. How do you know whether it’s the best choice? There’s a lot of information but digesting it might be difficult as it’s all deeply technical.

We’ll try our best to provide you with enough information to make a decision whether or not you should use Kurento in your video app. We’ll tell you why the media server is important, about the license, architecture, main functions that you can develop with Kurento, those being modules. We’ll finish with the summary of when it’s best to use a low-level media server, such as Kurento, and when it’s best to use an out-of-the-box solution.

WebRTC and why we need a media server

Kurento is an open-source WebRTC media streaming server with many built-in video conferencing modules released under the Apache license. WebRTC is a standardized, low latency, real-time, browser-to-browser transmission method without the need for third-party plugins or extensions. WebRTC is a fully client-side technology, so why would we need a media server?

The main reason is the load on the client with a large number of participants. The number of connections between participants grows exponentially, at the same time video quality worsens and the load on traffic and system resources grows. WebRTC can be used as normal P2P communication between 2-6 (in our experience), but if there are more participants, it makes more sense to use a media server. In addition, there are difficulties if we need to save a video recording to a separate file or somehow process it on the fly because all the work lies on the client.

WebRTC is also quite a secure option. Why? We explain here.

A short history of Kurento

Kurento was developed in 2010 in Madrid as a separate open-source project. The main language that Kurento uses is C++, which helps to optimize the resources of the system.

The media server has more than 2 900 stars on GitHub and a few hundred forks, which are separate branches of the project supported by the community.

At the moment Kurento development team joined Twilio and there are minor versions of Kurento itself with some minor patches, and new versions are still being released.

Apache, the license of Kurento

Kurento is released under the Apache license. It gives the developers absolute freedom when working with a code. You just have to mention what changes you apply and who the first author is.

Products with software under the Apache license can be used in commercial products. You can use Kurento in your products for free and get income from those products. No need to pay any loyalty.

You can find the full text about Apache here.

Kurento architecture: MCU and SFU

There are two main types of media server architectures: MCU and SFU. We will give an overview in this article. If you’re interested in more details, check out this article.

MCU (Multipoint Conferencing Unit) is a collage-like video architecture. We have multiple streams of users which make up one big seamless picture with each item’s location unchanged. The MCU takes an outgoing video stream from each participant, then the media server stitches all the streams into one with a fixed layout. Thus, despite the fact that there are many participants in the conference, each client receives only one stream as input. This allows to save CPU resources and traffic consumption on the client-side, but increases the load on the server itself and limits the possibilities of video chat layout customization. With MCU we can’t tell exactly which participant is on the video. Mixing streams requires a lot of processing power, increasing the cost of maintaining the server. This type of architecture is mostly suitable for meetings with a large number of participants (> 40). MCU is also a good solution if you need to get video streams on weak devices e.g. on the phone thanks to processing on the server.

SFU (Selective Forwarding Unit) is a popular architecture in modern WebRTC solutions that allows the video conference client to receive only the video streams it needs at the moment. SFU is more like a mosaic where you have to assemble the elements yourself, but the order of assembly is up to you. In SFU each participant also sends his stream to the server, but the other stream comes separately. This architecture better distributes the load between server and client and gives full control over the implementation of the video chat interface. Unlike MCU, a server with SFU doesn’t have to decode and transcode incoming streams. This helps to significantly reduce the load on the server CPU. SFU is well suited for broadcasting (one-to-many or one-person streaming) due to its ability to dynamically scale the system depending on the number of streams. At the same time, this type of server requires more outgoing server bandwidth because it has to do more streams to clients.

Let’s compare these 2 types using the 4-people video conference as an example.

On the client:

On the server:

There are also systems that use hybrid architectures to achieve the best result depending on the current number of users and the needs of a particular client. For example, if the client is a weak mobile device, it can receive a single stream from the media server as in the MCU. Browser users, on the other hand, will receive streams separately, where it is possible to implement a unique way of meshing elements as in SFU. Another case would be interaction with SIP devices (mostly IP telephony) where only the MCU is supported. Some hybrids may change from SFU to MCU on the fly when the number of participants reaches a certain threshold. There is the XDN (Experience Delivery Network) architecture from red5pro, which uses cloud technologies to solve WebRTC scaling problems. It has clusters that consist of different types of nodes. There are the source, relay, and edge nodes. Within this topology, any source node receives incoming streams and exchanges data with several edge nodes to support thousands of participants. For larger cases, source nodes can pass the flow to relay nodes, which pass the flow to multiple edge nodes to scale the cluster even more.

Kurento allows both types, using SFU by default. MCU architecture can be achieved using the Compositor element. You can mix SFU and MCU to obtain a hybrid type. Kurento uses SFU by default.

Description of the main modules and functionality

What are modules: main principles of Kurento design

Kurento includes several basic modules tailored to work not only with WebRTC, but also with video recording, computer vision, and AR filters. The media server would be a good choice if you want to work directly with regular WebRTC without using additional wrappers. It can be useful if you want to integrate your video chat with a native Android and iOS app. Kurento does not include a signaling mechanism, you can choose what works best for you depending on your project requirements, be it WebSockets or something else. Since Kurento is an open-source project, this significantly reduces the cost of using a media server.

Unlike the off-the-shelf paid media servers, Kurento is a rather low-level solution that allows you to customize the interaction of modules in the pipeline. Also, unlike, for example, Jitsi, which is a boxed solution and has a ready-made interface, with Kurento it’s much easier to choose and implement the interface you want. And of course, you can write your own module extending the standard media server functionality. The basis of the Kurento architecture design is modularity, where each media element is a program block that performs a certain task and can interact with other elements. In Kurento jargon, the developers call this Media Pipeline and Media Elements. Media Elements are simply modules that connect to each other within the Pipeline. Note, however, that the topology of the Pipeline is chosen by the developers, it does not necessarily have to be a linear sequence of elements.

Let’s talk about the main modules now.

WebRTC video conference

The basis of Kurento for video conferencing is the WebRtcEndpoint. Using it, we can transfer WebRTC streams and interconnect them in a single pipeline. In a video conference, the pipelines can usually be thought of as one room that is isolated from other similar rooms. In this case, WebRtcEndpoint is the user who wants to broadcast their own video or watch someone else’s.

Recording video and audio

The media server uses RecorderEndpoint to record. We can connect the WebRTC stream to the recorder, and the recorder will record the result to the file system. After connecting and starting the video just call the Kurento API method “record” of the recorder. We can specify restrictions when recording: for example, we can record only audio or only video.

Playing third-party media

Kurento has a built-in PlayerEndpoint module that allows you to play third-party audio and video files as well as RTMP and RTSP streams. This means that you can add a stream from the IP camera and broadcast it to other participants of the video conference. You can use the player to play music for users, which is good for music platforms, online training, or even text-to-speech. Thanks to the flexibility of the pipeline, it’s possible to play media not only to all participants but also to a specific user depending on your needs. You can do this simply by creating a new PlayerEndpoint and connecting it to the desired WebRtcEndpoint using the connect method. Once you need to play a media file, just call the player’s play method.

Computer vision and filters

The media server allows you to use many filters. For example, ZBarFilter is used to recognize QR codes, FaceOverlayFilter can recognize a face in a video stream and highlight it in real-time. And with GStreamerFilter, you can create your own custom filter. Also, Kurento has several experimental modules, including filters which are installed separately. There is kms-crowddetector that can detect a crowd, and there is a kms-platedetector that can detect a vehicle’s license plate number.

No support for multi-streaming with different resolutions

There is no support for Simulcast (sending multiple copies of the same stream in different quality) and SVC (sending a stream in low quality with the ability to add additional layers to improve quality if needed).

There’s a solution. You can create several streams and request videos with different resolutions. Switch between video streams depending on packet losses. However, this solution is far from ideal as it puts a huge load on the server. So let’s be honest: there’s no solution that’d be good per se.

Codec support in Kurento

In order to stream over a network, computers use codecs. Codecs are programs that can perform data or signal encoding. They allow you to compress video or audio to an acceptable size for network streaming and playback.

Kurento Media Server supports codecs like H.264, VP8/9 for video, and OPUS for audio. They allow you to achieve a high degree of compression of the video stream while maintaining high quality. These codecs are the current WebRTC standards. There is no support for AV1 and H.265. These are the latest standards that allow you to reduce the bitrate for the same quality as their older counterparts. The media server itself compresses the received video stream to a quality that matches the bandwidth to the client at that moment, and if necessary, transcodes the stream into the codec the client needs.

API, documentation, Typescript support

The Kurento API can be accessed via the kurento-client library and various additional libraries, e.g. kurento-utils. You can find detailed documentation on the official Kurento website with the description of every module and usage examples for Java and Javascript. you can go either here or here. The site provides information not only about the media server itself but also about related WebRTC themes, such as configuring the TURN (Traversal Using Relay NAT) server, necessary to bypass the communication limitations of users behind a symmetric NAT. The TURN protocol allows you to get the IP address and port needed to establish a WebRTC connection in the face of NAT restrictions. Symmetric NATs have additional protection against transport data tampering. The symmetric NAT table stores 2 more parameters – IP and port of the remote host. Packets from the external network are discarded because the source data doesn’t match the data recorded in the table. The client library supports types for Typescript.

OpenVidu

Based on Kurento, there is the OpenVidu framework, which is designed for simpler development if you don’t need anything other than simple video conferencing. No need to use a server to communicate with the framework, just connect the client library. The library has many wrappers for all popular frontend frameworks as well as for Android and iOS. OpenVidu is a free solution, but there is a paid version that provides additional features for monitoring and scaling the WebRTC platform. And there are also future plans for Simulcast and SVC and automatic switching to P2P sessions for 1-1 cases.

The number of participants in a video conference

Kurento developers have their own benchmark to determine the maximum number of sessions on one machine. Although it’s originally intended for OpenVidu, it’s also suitable for Kurento. Below you can see a table for the different machines on AWS.

video-conferencing-AWS-benchmarks — Video conferencing benchmarks for AWS machines

For example, a single AWS c5.large instance with two CPU cores and 4 GB of RAM can handle 4 group-type meetings with 7 participants. In each such meeting, users show and watch each other’s streams (many-to-many).

Summary: when is Kurento suitable?

for a video conferencing platform
for integration with AR,
for video recording
for playing media files for conference participants
for working with pure RTP, for example, for live streaming.

Kurento is a low-level media server. If you use Kurento, development might take a bit more time than out-of-the-box solutions. That’s if your developer doesn’t specialize in Kurento.

However, Kurento allows you to implement any functionality you want. It often happens with ready-made solutions that the customer asks to create something else, but that functionality isn’t supported.

If you’re interested in Kurento media server installation for your project, request a free quote, and we’ll get back to you ASAP!

Tags Kurento, WebRTC

Uncategorized

How to Make a Custom Call Notification on Android? With Code Examples

Post author By Dmitry P
Post date 09.09.2021
2 Comments on How to Make a Custom Call Notification on Android? With Code Examples

How to create a custom Android call notification

Channel creation (api 26+)
Displaying a notification
Button actions upon clicking
Notification.CallStyle (api 31+)
Notifications with their own design
Full-screen notifications
Conclusion

You will learn how to make incoming call notifications on Android from basic to advanced layouts from this article. Customize the notification screen with our examples.

Last time, we told you what any Android app with calls should have and promised to show you how to implement it. Today we’ll deal with notifications for incoming calls: we’ll start with the simplest and most minimalistic ones, and end with full-screen notifications with an off-system design. Let’s get started!

Channel creation (api 26+)

Since Android 8.0, each notification must have a notification channel to which it belongs. Before this version of the system, the user could either allow or disallow the app to show notifications, without being able to turn off only a certain category, which was not very convenient. With channels, on the other hand, the user can turn off annoying notifications from the app, such as ads and unnecessary reminders, while leaving only the ones he needs (new messages, calls, and so on).

If we don’t specify a channel ID, using the Deprecated builder. If we don’t create a channel with such an ID, the notification will not be displayed with the Android 8 or later versions.

We need the androidx.core library which you probably already have hooked up. We write in Kotlin, so we use the version of the library for that language:

dependencies { 
   implementation("androidx.core:core-ktx:1.9.0")
}

All work with notifications is done through the system service NotificationManager. For backward compatibility, it is always better to use the Compat version of Android classes if you have them, so we will use NotificationManagerCompat. To get the instance:

val notificationManager = NotificationManagerCompat.from(context)

Let’s create our channel. You can set a lot of parameters for the Let’s create our channel. You can set a lot of parameters for the channel, such as a general sound for notifications and a vibration pattern. We will set only the basic ones, and the full list you can find here.

val INCOMING_CALL_CHANNEL_ID = “incoming_call” 
 
// Creating an object with channel data 
 
val channel = NotificationChannelCompat.Builder( 
 
   // channel ID, it must be unique within the package 
 
   INCOMING_CALL_CHANNEL_ID, 
 
   // The importance of the notification affects whether the notification makes a sound, is shown immediately, and so on. We set it to maximum, it’s a call after all. 
 
   NotificationManagerCompat.IMPORTANCE_HIGH 
 
) 
 
   // the name of the channel, which will be displayed in the system notification settings of the application 
 
   .setName(“Incoming calls”) 
 
   // channel description, will be displayed in the same place 
 
   .setDescription(“Incoming audio and video call alerts”) 
 
   .build() 
 
// Creating the channel. If such a channel already exists, nothing happens, so this method can be used before sending each notification to the channel. 
 
notificationManager.createNotificationChannel(channel)

create-channel-notifcation-on-android — Notification channel in application settings

Notification runtime permission (api 33+):

If your app targets Android 13+ you should declare the following permission in AndroidManifest:

<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />

You should also request POST_NOTIFICATIONS permission from a user at runtime. Learn more about permission requesting.

Displaying a notification

Now we can start creating the notification itself, let’s start with the simplest example:

val notificationBuilder = NotificationCompat.Builder(  
 
this,  
 
   // channel ID again 
 
   INCOMING_CALL_CHANNEL_ID 
 
) 
 
   // A small icon that will be displayed in the status bar 
 
   .setSmallIcon(R.drawable.icon) 
 
   // Notification title 
 
   .setContentTitle(“Incoming call”) 
 
   // Notification text, usually the caller’s name 
 
   .setContentText(“James Smith”) 
 
   // Large image, usually a photo / avatar of the caller 
 
   .setLargeIcon(BitmapFactory.decodeResource(resources, R.drawable.logo)) 
 
   // For notification of an incoming call, it’s wise to make it so that it can’t be “swiped” 
 
   .setOngoing(true) 
 
       So far we’ve only created a sort of “description” of the notification, but it’s not yet shown to the user. To display it, let’s turn to the manager again: 
 
// Let’s get to building our notification 
 
val notification = notificationBuilder.build() 
 
// We ask the system to display it 
 
notificationManager.notify(INCOMING_CALL_NOTIFICATION_ID, notification)

set-up-display-android-notification — Simple notification

The INCOMING_CALL_NOTIFICATION_ID is a notification identifier that can be used to find and interact with an already displayed notification.

For example, the user wasn’t answering the call for a long time, the caller got tired of waiting and canceled the call. Then we can cancel notification:

notificationManager.cancel(INCOMING_CALL_NOTIFICATION_ID)

Or, in the case of a conferencing application, if more than one person has joined the caller, we can update our notification. To do this, just create a new notification and pass the same notification ID in the notify call — then the old notification will just be updated with the data, without animating the appearance of the new notification. To do this, we can reuse the old notificationBuilder by simply replacing the changed part in it:

notificationBuilder.setContentText(“James Smith, George Watson”) 
 
notificationManager.notify( 
 
   INCOMING_CALL_NOTIFICATION_ID,  
 
   notificationBuilder.build() 
 
)

Button actions upon clicking

A simple notification of an incoming call, after which the user has to find our application himself and accept or reject the call is not a very useful thing. Fortunately, we can add action buttons to our notification!

To do this, we add one or more actions when creating the notification. Creating them will look something like this:

val action = NotificationCompat.Action.Builder( 
 
   // The icon that will be displayed on the button (or not, depends on the Android version) 
 
   IconCompat.createWithResource(applicationContext, R.drawable.icon_accept_call), 
 
   // The text on the button 
 
   getString(R.string.accept_call), 
 
   // The action itself, PendingIntent 
 
   acceptCallIntent 
 
).build()

Wait a minute, what does another PendingIntent mean? It’s a very broad topic, worthy of its own article, but simplistically, it’s a description of how to run an element of our application (such as an activity). In its simplest form it goes like this:

const val ACTION_ACCEPT_CALL = 101 
 
// We create a normal intent, just like when we start a new Activity 
 
val intent = Intent(applicationContext, MainActivity::class.java).apply { 
 
   action = ACTION_ACCEPT_CALL 
 
} 
 
// But we don’t run it ourselves, we pass it to PendingIntent, which will be called later when the button is pressed 
 
val acceptCallIntent = PendingIntent.getActivity(applicationContext, REQUEST_CODE_ACCEPT_CALL, intent, PendingIntent.FLAG_UPDATE_CURRENT)

Accordingly, we need to handle this action in activity itself

To do this, in `onCreate()` (and in `onNewIntent()` if you use the flag `FLAG_ACTIVITY_SINGLE_TOP` for your activity), take `action` from `intent` and take the action:

override fun onNewIntent(intent: Intent?) { 
 
   super.onNewIntent(intent) 
 
   if (intent?.action == ACTION_ACCEPT_CALL)  
 
       imaginaryCallManager.acceptCall() 
 
}

Now that we have everything ready for our action, we can add it to our notification via `Builder`:

notificationBuilder.addAction(action)

In addition to the buttons, we can assign an action by clicking on the notification itself, outside of the buttons. Going to the incoming call screen seems like the best solution — to do this, we repeat all the steps of creating an action, but use a different action id instead of `ACTION_ACCEPT_CALL`, and in `MainActivity.onCreate()` handle that `action` with navigation:

override fun onNewIntent(intent: Intent?) { 
 
   … 
 
   if (intent?.action == ACTION_SHOW_INCOMING_CALL_SCREEN) 
 
       imaginaryNavigator.navigate(IncomingCallScreen()) 
 
}

Notification.CallStyle (api 31+):

There’s a new call notification style in Android 12, Notification.CallStyle. It helps you distinguish and highlight the call notification among others. In the new OS versions starting from Android 12 you have to use Notification.CallStyle instead of custom call notifications.

// Dividing the logic of creating notifications for Android 12+ below
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
   // Creating a notification with Notification.CallStyle

   val icon = Icon.createWithResource(this, R.drawable.user_avatar)
   val caller = Person.Builder()
       // Caller icon
       .setIcon(icon)
       // Caller name
       .setName("Chuck Norris")
       .setImportant(true)
       .build()

   // Creating the call notification style
   val notificationStyle = Notification.CallStyle.forIncomingCall(caller, declineIntent, answerIntent)

   Notification.Builder(this, CHANNEL_ID)
       .setSmallIcon(R.drawable.ic_launcher_foreground)
       .setContentTitle("Incoming call")
       .setContentText("Incoming call from Chuck Norris")
       .setStyle(notificationStyle)
       // Intent that will be called for when tapping on the notification
       .setContentIntent(contentIntent)
       .setFullScreenIntent(contentIntent, true)
       .setOngoing(true)
       // notification category that describes this Notification. May be used by the system for ranking and filtering
       .setCategory(Notification.CATEGORY_CALL)
       .build()
} else {
   // Creating the custom notification
   ...
}

Notification with Notification.CallStyle

Notifications with their own design

Notifications themselves are part of the system interface, so they will be displayed in the same system style. However, if you want to stand out, or if the standard arrangement of buttons and other notification elements don’t suit you, you can give the notifications your own unique style.

DISCLAIMER: Due to the huge variety of Android devices with different screen sizes and aspect ratios, combined with the limited positioning of elements in notifications (relative to regular application screens), Custom Content Notification is much more difficult to support

The notification will still be rendered by the system, that is, outside of our application process, so we need to use RemoteViews instead of the regular View. Note that this mechanism does not support all the familiar elements, in particular, the `ConstraintLayout` is not available.

A simple example is a custom notification with one button for accepting a call:

<!– notification_custom.xml –> 
 
<RelativeLayout  
 
   … 
 
   android:layout_width=”match_parent” 
 
   android:layout_height=”match_parent”> 
 
   <Button 
 
       android:id=”@+id/button_accept_call” 
 
       android:layout_width=”wrap_content” 
 
       android:layout_height=”wrap_content” 
 
       android:layout_centerHorizontal=”true” 
 
       android:layout_alignParentBottom=”true” 
 
       android:backgroundTint=”@color/green_accept” 
 
       android:text=”@string/accept_call” 
 
       android:textColor=”@color/fora_white” />

</RelativeLayout>.

The layout is ready, now we need to create an instance RemoteViews and pass it to the notification constructor

val remoteView = RemoteViews(packageName, R.layout.notification_custom) 
 
// Set the PendingIntent that will “shoot” when the button is clicked. A normal onClickListener won’t work here – again, the notification will live outside our process 
 
remoteView.setOnClickPendingIntent(R.id.button_accept_call, pendingIntent) 
 
// Add to our long-suffering builder 
 
notificationBuilder.setCustomContentView(remoteView)

create-custom-android-notification — Notification with custom layout

Our example is as simplistic as possible and, of course, a bit jarring. Usually, a customized notification is done in a style similar to the system notification, but in a branded color scheme, like the notifications in Skype, for example.

In addition to .setCustomContentView, which is a normal notification, we can separately specify mark-up for the expanded state .setCustomBigContentView and for the head-up state .setCustomHeadsUpContentView

Full-screen notifications

Now our custom notification layouts match the design inside the app, but they’re still small notifications, with small buttons. And what happens when you get a normal incoming call? Our eyes are presented with a beautiful screen that takes up all the available space. Fortunately, this functionality is available to us! And we’re not afraid of any limitations associated with RemoteViews, as we can show the full `activity`.

First of all, we have to add a permission to `AndroidManifest.xml`

<uses-permission android:name="android.permission.USE_FULL_SCREEN_INTENT" />

After creating an `activity` with the desired design and functionality, we initialize the PendingIntent and add it to the notification:

val intent = Intent(this, FullscreenNotificationActivity::class.java) 
 
val pendingIntent = PendingIntent.getActivity(applicationContext, 0, intent, PendingIntent.FLAG_UPDATE_CURRENT) 
 
// At the same time we set highPriority to true, so what is highPriority if not an incoming call? 
 
notificationBuilder.setFullScreenIntent(pendingIntent, highPriority = true)

And that’s it! Despite the fact that this functionality is so easy to add, for some reason not all call-related applications use it. However, giants like Whatsapp and Telegram have implemented notifications of incoming calls in this way!

create-full-screen-android-notification — Custom incoming call screen

Bottom line

The incoming call notification on Android is a very important part of the application. There are a lot of requirements: it should be prompt, eye-catching, but not annoying. Today we learned about the tools available to achieve all these goals. Let your notifications be always beautiful!

Read other articles from this cycle:

How to Make Picture-in-Picture Mode on Android With Code Examples

How to Implement Audio Output Switching During the Call on Android App?

How to Implement Foreground Service and Deep Links for Android apps with calls?

Liked this article? Then the next step is learning how to make a video chat on Android. An explanation on how to do it for beginners is here.

Tags Android, mobile app development, WebRTC