Categories
Uncategorized

How to implement Picture-in-Picture mode in React.JS

implement-picture-in-picture

Tutorial on how to set up PiP
What about Firefox?
Software PiP activation
Check support
Opening and closing functions
Automatic activation of web PiP
PiP controls
Linking with video conferencing
The pitfalls
Conclusion with a bonus!

Picture-in-picture (PIP) is a separate browser window with a video that sits outside the page.

You minimize the tab or even the browser where you’re watching the video, and it’s still visible in the little window. It’s handy if you’re broadcasting a screen and want to see your interlocutors or are watching a really interesting show, but someone sent a message.

picture-in-picture-youtube
That’s how Youtube picture-in-picture mode looks like

Let’s figure out how to create this window and make it work.

As of early 2022, the Picture-in-picture specification is in draft form. All browsers work differently, and the support leaves a lot to be desired.

As of early 2022, only 48% of browsers support the feature.

Let’s go to a technical guide to implementation with all the pitfalls and unexpected plot twists. Enjoy 🙂

Tutorial on how to set up PiP

First, you need a video element.

<video controls src="video.mp4"></video>

In Chrome and Safari, the PiP activation button should appear. 

picture-in-picture-chrome
In Chrome, click on the 3 dots in the bottom right corner

5 minutes and it’s done! 

What about Firefox?

Unfortunately, Mozilla doesn’t yet have full picture-in-picture support 🙁

To activate PiP in Mozilla, every user has to go into configuration (type about:config in the search box). Then find media.videocontrols.picture-in-picture.enabled and make it true.

Because of the weak support for PIP in Mozilla, we won’t look at that browser any further.

Now you can activate web picture-in-picture in all popular browsers. 

But what if this is not enough?

Could it be more convenient?

Maybe add a nice activation button?

Or automatically switch to PIP when you leave the page? 

Yes, this is all possible! 

Software PiP activation

To start, let’s implement the basic open/close functionality and connect the button.

Let’s say your browser supports picture-in-picture on the web. To open and close the PiP window, we need to:

  1. make sure the feature is supported
  2. make sure that there is no other PIP 
  3. implement the cross-browser picture-in-picture activation/deactivation function.

Check support

To make sure we can programmatically activate a PIP window, we need to know if it is activated in the browser and if there is an opening method.

You can check the activation status through the document pictureInPictureEnabled property:

"pictureInPictureEnabled" in document && document.pictureInPictureEnabled

To make sure that we can interact with the PIP window, let’s try to find a picture-in-picture activation method.

For Safari it’s webkitSetPresentationMode, for all other browsers requestPictureInPicture.

xport const canPIP = (): boolean => "pictureInPictureEnabled" in document &&
 document.pictureInPictureEnabled;

const supportsOldSafariPIP = () => {
 const video = document.createElement("video");

 return (
   canPIP() &&
   video.webkitSupportsPresentationMode &&
   typeof video.webkitSetPresentationMode === "function"
 );
};

const supportsModernPIP = () => {
 const video = document.createElement("video");

 return (
   canPIP() &&
   video.requestPictureInPicture &&
   typeof video.requestPictureInPicture === "function"
 )
};

const supportsPIP = (): boolean => supportsOldSafariPIP() || supportsModernPIP(

Checking for the presence of a PiP window

To determine whether or not we already have a picture-in-picture window, you can look it up in the properties of the document

document.pictureInPictureElement

Opening and closing functions

Opening function

The standard requestPictureInPicture open method.

video.requestPictureInPicture();

For more support among browsers, let’s implement a fallback. To enter picture-in-picture on safari, you need to use the webkitSetPresentationMode method of the video element:

video.webkitSetPresentationMode("picture-in-picture")

Closing function

The standard closing method:

document.exitPictureInPicture()

Fallback for Safari

video.webkitSetPresentationMode("inline")

As a result, we have the functionality to open or close the PIP.

export const canPIP = () =>
 "pictureInPictureEnabled" in document &&
 document.pictureInPictureEnabled;

const isInPIP = () => Boolean(document.pictureInPictureElement);

const supportsOldSafariPIP = () => {
 const video = document.createElement("video");

 return (
   canPIP() &&
   video.webkitSupportsPresentationMode &&
   typeof video.webkitSetPresentationMode === "function"
 );
};

const supportsModernPIP = () => {
 const video = document.createElement("video");

 return (
   canPIP() &&
   video.requestPictureInPicture &&
   typeof video.requestPictureInPicture === "function"
 )
};
const supportsPIP = () =>
 supportsOldSafariPIP() || supportsModernPIP();

export const openPIP = async (video) => {
 if (isInPIP()) return;

 if (supportsOldSafariPIP())
   await video.webkitSetPresentationMode("picture-in-picture");
 if (supportsModernPIP())
   await video.requestPictureInPicture();
};

const closePIP = async (video) => {
 if (!isInPIP()) return;

 if (supportsOldSafariPIP())
   await video.webkitSetPresentationMode("inline");
 if (supportsModernPIP())
   await document?.exitPictureInPicture();
};

Now, all we have to do is enable the button.

const disablePIP = async () => {
 await closePIP(videoElement.current).catch(/*handle error*/)
};

const enablePIP = async () => {
 await openPIP(videoElement.current).catch(/*handle error*/)
};

const handleVisibility = async () => {
 if (document.visibilityState === "visible") await disablePIP();
 else await enablePIP();
};

const togglePIP = async () => {
 if (isInPIP()) await disablePIP()
 else await enablePIP()
};

Don’t forget to catch errors from asynchronous functions and connect the functionality to the button.

<button onClick={togglePIP} className={styles.Button}>
 {isPIPOn ? "Turn off PIP" : "Turn on PIP"}
</button>
floating-window-video-player
How to open and close pip mode in a browser?

See? Not so much code and the button for switching between PiP and normal mode is ready!

Automatic activation of web picture-in-picture

Why do you need picture-in-picture?

To surf the Internet and watch video streams from another page!

Chatting in a video conference in your browser, you want to tell something while peeking into Google Docs but still seeing the person you’re talking to, just like in Skype. You can do that with PiP. Or you want to keep watching the movie while answering an urgent message in a messenger – this is also possible if the site where you watch the movie has developed PiP functionality.

Let’s implement the automatic opening of the PiP window when you leave the page.

Safari has the autoPictureInPicture property, it turns on the Picture-In-Picture mode only if the user is watching a fullscreen video.

To activate it, you need to make the video element property autoPictureInPicture true.

if (video && "autoPictureInPicture" in video) {
  video.autoPictureInPicture = true;
}

That’s it for Safari.

Chrome and similar browsers allow you to ping without a fullscreen, but the video must be visible and the focus must be on the page.

You can use the Page Visibility API to track page abandonment.

document.addEventListener("visibilitychange", async () => {
 if (document.visibilityState === "visible")
   await closePIP(video);
 else
   await openPIP(video);
});

Enjoy, the picture-in-picture auto-activation is ready.

PIP Controls

PiP video has the following buttons by default:

  • pause (except when we pass a media stream to a video tag)
  • switch back to the page 
  • next/previous video

Use the media session API to configure video switching.

navigator.mediaSession.setActionHandler('nexttrack', () => {
 // set next video src
});
navigator.mediaSession.setActionHandler('previoustrack', () => {
 // set prev video src
});
video-player-mini-view
Customised picture-in-picture mode

Let’s say we want to make a browser-based Skype with screen sharing.

It would be nice to show the demonstrator’s face. And also so that he can see himself, should, for example, his hair end up disheveled.

Javascript picture-in-picture would be perfect for that!

To display a WebRTC media stream in PiP, all you have to do is apply it to the video, and that’s it.

video.srcObject = await navigator.mediaDevices.getUserMedia({
 video: true,
 audio: true,
})
video-pop-up-player
Implement picture-in-picture mode for video calls

In this uncomplicated way, you can show the face of the screen demonstrator. And best of all, there is no need to transmit additional video of the speaker’s face, because it’s already present in the demonstration exactly where the author wishes it to be.

This not only saves traffic for all users in the video conference but also creates a more convenient interface for the demonstrator and the audience.

The same logic works with the interlocutor in an online conference.

Anything that can be displayed in the video tag can be displayed in the PiP window.

The pitfalls

Nothing works perfectly from the first try 🙂 Here are some tips on what to do when picture in picture mode is not working.

Error: Failed to execute ‘requestPictureInPicture’.

DOMException: Failed to execute ‘requestPictureInPicture’ on ‘HTMLVideoElement’: Must be handling a user gesture if there isn’t already an element in picture-in-picture JS.

So either the browser has realized that we’re abusing the API, or you forgot to check if the window is open

In the w3 draft, the requirements are userActivationRequired and playingRequired. This means that picture-in-picture can only be activated when the user interacts and if the video is playing.

At the moment the error can be found in 2 popular cases: 

  • (Chrome) trying to navigate to PiP if the page is out of focus.
  • (Safari) attempt to navigate to PiP without user interaction 

The video in the PiP window doesn’t update

To deal with this problem in react, just change the key property along with the media stream update or src.

<video controls key={/* updated key */} src="video.mp4"></video>

Video in the PiP window freezes

From time to time a video hangs. This usually happens when the video tag disappears from the page. In such a situation, you need to call the document.exitPictureInPicture() method.

When starting a broadcast in another tab or application, the auto-opening PiP window doesn’t work (Chrome)

This problem is related to this error. The reason is that when you click on the system window to select a tab or page to show, our page loses focus. If there is no focus, the userActivationRequired condition can’t be satisfied, so you can’t open Pip right after the start of the demonstration.

However, it is possible to open a PiP window in advance, say, when the page loses focus:

document.addEventListener("blur", () => {
 // open PIP
})

In this case, the PiP will open before the broadcast begins.

Conclusion

Despite pretty weak browser support, only 48% as of early 2022, Javascript-enabled PiP is a pretty quick feature to implement and brings an amazing user experience to web app users with video or broadcasts.

However, you should consider the fact that half of the users may never use it due to poor support.

You can test this feature out in the sandbox.

BONUS TIP

How to turn on picture-in-picture on YouTube?

  1. Turn on the video
  2. Open console. For macOS, use Option + ⌘ + J. For Windows or Linux, use Shift + CTRL + J.
  3. Enter this code:
document.onclick = () => {
 document.querySelector('video')?.requestPictureInPicture();
 document.onclick = null;
};

4. Press Enter.
5. Click on an empty spot on the page.

Categories
Uncategorized

How to Make Picture-in-Picture Mode on Android With Code Examples

This is what Picture-in-Picture mode looks like

In recent years, smartphones have become increasingly close to computers in terms of functionality, and many are already replacing the PC as their primary tool for work. The advantage of personal computers was multi-window capability, which remained unavailable on smartphones. But with the release of Android 7.0, this began to change and multi-window support appeared.

   It’s hard to overestimate the convenience of a small floating window with the video of the interlocutor when the call is minimized – you can continue the dialogue and simultaneously take notes or clarify some information. Android has two options for implementing this functionality: support for the application in a floating window and a picture-in-picture mode. Ideally, an application should support both approaches, but the floating window is more difficult to develop and imposes certain restrictions on the overall application design, so let’s consider picture-in-picture (PiP) on Android as a relatively simple way to bring multi-window support into your application.

pop up video call
PIP mode for video calls on Android

Switching to PIP mode

        Picture-in-picture on Android is supported for most devices in the 8 version and above. Accordingly, if you support system versions lower than this, all PIP mode-related calls should be wrapped in the system version check:

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) { 
    // Something related to PiP 
}

   The entire `Activity` is converted to PIP, and first, you need to declare PIP support for this `Activity` in `AndroidManifest.xml`:

<activity
    ...
    android:supportsPictureInPicture="true" />

       Before using picture-in-picture it is necessary to make sure that the user’s device supports this mode, to do this we turn to the `PackageManager`.

val isPipSupported = context.packageManager.hasSystemFeature(PackageManager.FEATURE_PICTURE_IN_PICTURE)

After that, in its simplest form, the transition to picture-in-picture mode is done literally with one line:

this.enterPictureInPictureMode()

   But to go to it, you need to know when it is convenient for the user. You can make a separate button and jump when you click on it. The most common approach is an automatic switch when the user minimizes the application during a call. To track this event, there is a handy method `Activity.onUserLeaveHint` called whenever the user intentionally leaves `Activity` — whether via the Home or Recent button.

override fun onUserLeaveHint() {
    ...
    if (isPipSupported && imaginaryCallManager.isInCall)
        this.enterPictureInPictureMode()
}

Interface adaptation

        Great, now our call screen automatically goes into picture-in-picture on Android! But there are often “end call” or “change camera” buttons, and they will not work in this mode. It’s better to hide them when transitioning.

        To track the transition to / from PIP mode, `Activity` and `Fragment` have a method `onPictureInPictureModeChanged`. Let’s redefine it and hide unnecessary interface elements

override fun onPictureInPictureModeChanged(
    isInPictureInPictureMode: Boolean,
    newConfig: Configuration?
) {
    super.onPictureInPictureModeChanged(isInPictureInPictureMode, newConfig)
    setIsUiVisible(isInPictureInPictureMode)
}

   The PIP window is quite small, so it makes sense to hide everything except the interlocutor’s video, including the local user’s video — it will be too small to see anything there anyway.

How to implement picture-in-picture mode on Android app?

Customization

        The PIP window can be further customized by passing `PictureInPictureParams` in a call to `enterPictureInPictureMode`. There are not many customization options, but the option to add buttons to the bottom of the window deserves special attention. This is a nice way to keep the screen interactive despite the fact that the regular buttons stop working when the user activates the PIP mode.

        The maximum number of buttons you can add depends on many factors, but you can always add at least three. All buttons over the limit simply won’t be shown, so it’s better to place the especially important ones at the beginning. You can find out the exact limit in the current configuration through the method `Activity`:

this.maxNumPictureInPictureActions

        Let’s add an end call button to our PIP window. To start with, just like with notifications, we need a `PendingIntent`, which will be responsible for telling our application that the button has been pressed. If this is the first time you’ve heard of `PendingIntent’ — you can learn more about them in our last article.

        After that, we can start creating the actual button description, namely `RemoteAction`.

val endCallPendingIntent = getPendingIntent()
val endCallAction = RemoteAction(
    // An icon for a button. The color will be ignored and changed to a system color
    Icon.createWithResource(this, R.drawable.ic_baseline_call_end_24),
    // Text of the button that won't be shown
    "End call",
    // ContentDescription для screen readers
    "End call button",
    // Our PendingIntent  that'll be launched upon pressing the button
    endCallPendingIntent
)

        Our “action” is ready, now we need to add it to the PIP parameters and, subsequently, to the mode transition call.

        Let’s start by creating a Builder for our customization parameters:

val pipParams = PictureInPictureParams.Builder()
    .setActions(listOf(endCallAction))
    .build()

this.enterPictureInPictureMode(pipParams)
multi-window mode
How to customize picture-in-picture mode?

In addition to the buttons, through the parameters, you can set the aspect ratio of the PIP features on Android or the animation of switching to this mode.

Other articles about calls on Android

WebRTC on Android

How to Make a Custom Call Notification on Android? With Code Examples

What Every Android App With Calls Should Have

How to Implement Audio Output Switching During the Call on Android App?

How to Implement Foreground Service and Deep Links for Android apps with calls? With Code Examples

    Conclusion

        We have considered a fairly simple but very handy variant of using the multi-window feature to improve the user experience, learned how to add buttons to the PIP window on Android, and adapt our interface when switching to and from this mode.