Conference Webcast Report

Conversations around originating a webcast for conference, including cameras and mixers.
craiggsmith
Senior Member
Posts: 711
Joined: Sun Sep 12, 2010 2:14 pm
Location: South Jordan, Utah

Conference Webcast Report

Postby craiggsmith » Sun Oct 09, 2016 8:42 pm

We set up our broadcast to use the West US servers and the test went fine last night. This morning we were unable to get it working reliably. After several settings changes and restarts we called support and they said the west servers were having issues. They said to cancel the event and set up a new one using the east servers. The documentation says it takes up to 20 minutes for the server to be ready but they said it should only take a couple minutes. We waited a while but finally gave up and switched to the legacy webcast system as the meeting was ready to start. I checked back several times during the meeting and the new event never did say it was ready. Maybe if it doesn't start before the actual meeting start time it never will?

A few questions:
Why do we provide our contact info if they aren't going to use it to tell us there is a problem?
Why do we even have to specify a server?
Should we set up two events on two different servers just in case? Will it even let us do this?
Craig
STS
South Jordan, UT

russellhltn
Community Administrator
Posts: 20762
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Re: Conference Webcast Report

Postby russellhltn » Sun Oct 09, 2016 10:16 pm

craiggsmith wrote:the west servers were having issues.

I'll take a guess that there was another multi-stake conference, and several decided to re-broadcast it.

My question would be is if all the events were properly pre-scheduled, why couldn't the servers handle the load? Why weren't some schedules refused or moved to another server farm? Or was this some unforeseen external problem?
Have you searched the Wiki?
Try using a Google search by adding "site:tech.lds.org/wiki" to the search criteria.

lajackson
Community Moderators
Posts: 6140
Joined: Mon Mar 17, 2008 9:27 pm
Location: US

Re: Conference Webcast Report

Postby lajackson » Mon Oct 10, 2016 8:06 am

craiggsmith wrote:[Support] said to cancel the event [using the west servers] and set up a new one using the east servers.

We set up our stake conference event almost two weeks ago on the Southwest US servers. I was nervous because in two years we have never connected on conference Sunday without a call to the GSC. In each of the last three conferences, they determined that they needed to change a setting at their end, which they did, and we ultimately connected. I never had to speak with the GSC this time.

Yesterday we started up and connected with no problem around 8:15 am. At 9:27 am, the network cratered. But the VidiU reestablished the connection and restarted on its own. Two broadcast locations and a third monitoring location were able to reload the page and start receiving the broadcast feed again. After that, we had no troubles (attributable to the streaming servers) during the actual conference session that began at 10:00 am.

I was also nervous because of the horror stories on days where there is a regional stake conference broadcast. Because the satellite has been on "Not Authorized" all week (and there was not a temple dedication scheduled yesterday), I was afraid there might be a regional broadcast as russellhltn guessed. There may have been (I don't know), but it seems those stakes were not on our server farm (SW US) so we were not affected.

And yes, craiggsmith, we found during testing that if you set up a test event to begin prior to the present time, it never spins up. I would expect that to be true for an actual broadcast event, as well. We also found that this time includes the 10 minutes allotted before the actual test begins. So when we wanted to set up an unplanned test and it was 7:30 pm, we set a test event to begin at 7:42 pm. This allowed two minutes for the server to catch the event start and then begin to spin up for the 10-minute pre-test time at 7:32 pm. During weeknight testing, the servers seemed to spin up for a previously unscheduled test in about 2-3 minutes.

I would guess that if it were around 9:40 am and you were forced to set up an actual broadcast event, you would want to set the event to start at 11:45 am. The servers would see the two-hour preconference test time should begin at 9:45 am, spin up, and tell you they were good to go.

Somehow, the servers need to be triggered to spin up, and I think if you schedule an event and the time already has passed when they should be available, they never spin up.

And, of course, on a Sunday morning with an actual conference about to start, I doubt I would have the self-control to factor in an extra two minutes for the servers to trip. In fact, I probably would not even be able to do the math properly.

When I got home Sunday afternoon, I felt as it I had been standing on top of a cliff, without a railing, looking over the edge into the valley below, trying not to be knocked off balance by the wind gusts.

User avatar
seanhyte
Church Employee
Church Employee
Posts: 21
Joined: Tue Feb 16, 2016 12:49 pm
Contact:

Re: Conference Webcast Report

Postby seanhyte » Mon Oct 10, 2016 8:38 am

craiggsmith wrote:After several settings changes and restarts we called support and they said the west servers were having issues.

Interesting you had issues on the West server. We had our stake conference this weekend in Herriman, UT and I set it on the West server as well. We had 100% health the whole time on both the broadcast and receiving sites.

lajackson
Community Moderators
Posts: 6140
Joined: Mon Mar 17, 2008 9:27 pm
Location: US

Re: Conference Webcast Report

Postby lajackson » Mon Oct 10, 2016 8:54 am

seanhyte wrote:We had 100% health the whole time on both the broadcast and receiving sites.

And this is why I have so little hair left to pull out.

Where do you start to troubleshoot? Were the servers having problems as the help desk said? Was it the Internet bandwidth at the sending or receiving site? Was it the capability of the laptop that was used to receive, or a troublesome setting on the VidiU (I assume) that was used to originate?

Will the logs of the event provide answers or guidance to these questions if they are read by a skilled and knowledgeable person?

Our stake conference sported 100% health at every location. And the graphs show that one receiving location was solid as a rock.

But another receiving location looked like it had 27 heart attacks during the broadcast. And the graph of the originating location looks like it tried to crater every time I adjusted the camera.

I will agree that since we were able to connect on time and nothing ever buffered (that we noticed) we had a "successful" webcast. I would sleep better at night if I knew why the one receiving location was straight and steady in the graphs and the others were not.

User avatar
Mikerowaved
Community Moderators
Posts: 3132
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Re: Conference Webcast Report

Postby Mikerowaved » Mon Oct 10, 2016 3:34 pm

I'll be webcasting my stake conference this Sunday using my own system instead of the VidiU box. I'll let you know how things go.
So we can better help you, please edit your Profile to include your general location.

rannthal
Church Employee
Church Employee
Posts: 92
Joined: Mon Jun 09, 2014 1:41 pm

Re: Conference Webcast Report

Postby rannthal » Tue Oct 11, 2016 8:44 am

Alright everyone, calm down.

Last weekend there were unexpected license issues that caused some servers to not be able to spin up. Of course since these licenses were for third party software, it took some time for this to be corrected.

craiggsmith wrote:A few questions:
Why do we provide our contact info if they aren't going to use it to tell us there is a problem?
Why do we even have to specify a server?
Should we set up two events on two different servers just in case? Will it even let us do this?


The reason why contact information is wanted so support can contact the STS if there are problems. However, this has not been utilized very efficiently. We are taking steps to correct this and make it better both for our support engineers and STS. These will be added on our next release this coming month.

Since all this is using the cloud, there are data centers that are used. You specify the server so that you can receive the closest center to your location to minimize any congestion and jumps.

You should definately not set up two events. This would cause 2 servers to spin up, one which would not be used. When a server is spent up, it costs money. Not using a server is wasting the widow's mite.


A server is spun up 4 hours before the start time of the event. The first 2 hours are for support to catch any errors that the servers may have. (As I said before, this weekend there were third party license issues that took time to resolve.) The last 2 hours are for the STS to test there setup and location and correct any issues that are found there.

If you are having trouble troubleshooting, here is what I would do and have done in the past with my stake conferences in my stake.
1. Make sure that the laptops used are up to date and has at the very least 2 cores. Four cores are recommended.
2. Test the upload bandwidth at your broadcasting site. Test again and again. Get a good feel on what your internet upload is. Watch to see if there are any drops in the Internet and take note on how far they dropped. With all that, set your encoder to half of what your top upload speed is. In doing so you will minimize the risk of your Internet bandwidth to drop at the encoding bitrate. NOTE**** If you encoding site goes down or has problems then your receiving sites have no chance and will fail.)
3. Test the download speed at all your receiving sites. If there are sites that have low speed, disable the adaptive piece of the client player and set it to play at a low speed. This will cause to player to play only at that speed.

If you perform these above steps, you will decrease any issues with your events.

TEST, TEST, and TEST again. Know your system inside and out.

Schedule your event well into the future. Give yourself time to troubleshoot and test.

If you schedule your event on the day of the event and haven't tested anything, you are asking for problems. If you schedule an event to start with the next hour, you are short fusing the system. While this does work, it can cause problems.

There are instructions out on the tech system site. I realize they are out of date. We are working on revisions currently. But for the most part, if followed they will help.

If you have questions, ask. Contact me, I'll be glad to help and answer any questions you have about the system, the VidiU, how to read and interpret the graphs, etc. I have posted many tricks, tips, and help on this forum. Search it.

Lastly, do not think that we are not aware of problems that occur or that we don't care what is going on. We are working to make the system simpler, easier to use and more effective. This is only the 2nd year of this system's existence, so there are going to be going pains. Most of us eat our own work, meaning we are using the system for our own events. I am in fact the STS of my stake. I use the very system that I work on.

rannthal
Church Employee
Church Employee
Posts: 92
Joined: Mon Jun 09, 2014 1:41 pm

Re: Conference Webcast Report

Postby rannthal » Tue Oct 11, 2016 9:28 am

One more thing......about servers, here are some facts for the current system.

*Servers are located in data centers found in different regions throughout the world, utilizing the cloud.
*Servers are of different sizes pending on the size of the event.
*Servers are triggered to be spun up from the start time of the event.
*Servers take about 12 - 15 minutes to get spun up, licensed, and readied for an event.
*Any encoding to the server during the time of setup will be rejected.
*The system will tell you when a server is ready to be encoded to.
*If an event is set to start under the 15 minute mark can cause problems with the server being readied for the event. (The event starts before the server is ready, sometimes the server will not catch the actual go time.)
*After an event is held, a server is not shutdown immediately. This is to be able utilize any short fuse events that may be scheduled. (When a server is spun up, it costs money. The less servers spun up, the less of the widow's mite is spent. This is also why it seems that sometimes servers are available immediately. You got lucky enough to hit one of these servers....and no, there is no way to tell if this will be the case. Schedule events properly giving time for spin-up.)

User avatar
Mikerowaved
Community Moderators
Posts: 3132
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Re: Conference Webcast Report

Postby Mikerowaved » Sun Oct 16, 2016 1:59 pm

Mikerowaved wrote:I'll be webcasting my stake conference this Sunday using my own system instead of the VidiU box. I'll let you know how things go.

I'm pleased to report our Stake Conference webcasting went very smooth this morning using YouTube Live. The church currently has YouTube unblocked, which took away the necessity of bypassing the firewall in our two buildings. The video was uploaded in 720p using Wirecast, which was displayed in an overflow building using one of the stake's newer Epson HD projectors.
So we can better help you, please edit your Profile to include your general location.

craiggsmith
Senior Member
Posts: 711
Joined: Sun Sep 12, 2010 2:14 pm
Location: South Jordan, Utah

Re: Conference Webcast Report

Postby craiggsmith » Sun Oct 23, 2016 2:53 pm

Thanks everyone for the comments. I didn't realize that's how the system worked. Too bad the help desk wasn't familiar with this.

The weak link then seems to be the fact that we are pre-selecting a specific server instead of using a load balancer or something similar as is the norm.
Craig
STS
South Jordan, UT


Return to “Webcasting”

Who is online

Users browsing this forum: No registered users and 1 guest