Automatic highlights for any live video stream.

We saw that during live streams, many people want to see instant highlights right after a moment happens. Or they would like to just view the highlights of a stream but sometimes that isn't always available. In addition, we noticed that the audience reaction for most live streams is directly connected to the events occurring in the stream. If anything controversial or eventful happens at any moment, the audience starts reacting either positively or negatively. For example, consider a basketball game. Whenever a team scores a basket or the referee calls a foul, audience reactions start pouring in. We thought we could capture this relationship between audience reactions and events in the live stream to generate automatic highlights of the live stream.


To begin, we looked at Twitch, a popular streaming website that focuses on video game live streaming, including broadcasts of eSports competitions, in addition to creative content. We made a small web app to compliment this. The user can type in the address of a channel they are interested in, it will start looking at the stream to see if there is a highlight. If there is one, it will record it, and then make it available as a download.

We noticed that the chat for a certain streamer was closely tied to their stream.We generated automatic highlights of any live stream based on the associated density of chat comments per second. We maintained a moving average of the density of comments and if the chat exploded in comments, we would start recording. Once the chat died down, we would end the recording, add 8-7 seconds of footage before the highlight started and about 5 seconds of footage after the highlight ended and that would be a highlight.

We used StreamLighter on a Twitch stream to create the above video. As we look at moments 0:02 and 0:05, we notice that watchers begin to pay the streamer indicating that these moments are interesting and worth capturing.

What's next?

This approach can be applied to other streams beyond just video games. In cases where an accompanying chat is not available, we could capture reaction density via other means like social media. For example, to generate a highlight of a popular speech, we could actively track reactions on Twitter and capture moments where the number of tweets suddenly increase.

Built with

javascript, node.js, flask, python, azure, html/css