Skip to main content

Improper ‘provisioning and planning’ blamed for Apple’s botched live stream, not CDN capacity

lead_large

If you tuned in for Apple’s majorly hyped live stream of its iPhone 6/Apple Watch event yesterday, you probably noticed a few hiccups along the way: music playing from the Flint Center while another song streamed over the video, a translator speaking over much of the presentation, and the video just plain failing several times along the way. Dan Rayburn at StreamingMediaBlog.com shares his insight of what caused Apple’s major announcement video stream to botch itself throughout the show. According to Rayburn, it’s surprisingly not due to the number of viewers:

The bottom line with this event is that the encoding, translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes all didn’t work properly together and was the root cause of Apple’s failed attempt to make the live stream work without any problems. So while it would be easy to say it was a CDN capacity issue, which was my initial thought considering how many events are taking place today and this week, it does not appear that a lack of capacity played any part in the event not working properly. Apple simply didn’t provision and plan for the event properly.

Shortly before the event kicked off, many viewers were hit with the “TV Truck Schedule” image instead of Apple CEO Tim Cook welcoming the audience and jumping right into the new iPhone 6/iPhone 6 Plus news. Refreshing was either met with an ‘access denied’ page, a frozen stream, or the now famous truck schedule photo. For me, this was the behavior across three different screens, and others like the breakfast restaurant chain Denny’s Diner took note too.

[tweet https://twitter.com/DennysDiner/status/509396332843646977/]

Notably, Apple did accompany its live stream footage with a blog updated in real time with marketing information, product details, and tweets from celebrities and reporters, which helped fill in for the botched stream. At any rate, if the stream’s wonky behavior did come down to improper planning on the front end of the event with caching issues and S3 storage configuration, that hopefully means we won’t see a similar episode in the future.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Mac — experts who break news about Apple and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Mac on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel

Comments

  1. Steven Moore (@Stniuk) - 10 years ago

    Yes sat down to watch the event on the Apple TV and it was completely unwatchable due to fall outs, freezes and even the apple tv crashing. Terrible. I seen it later but it ruined the event.

    • originaldub - 10 years ago

      Sh*t happens. At least you “seen” it later.

      • vkd108 - 10 years ago

        This usage of “seen” is according to the new, dumbed-down book of grammar.

  2. I had to watch the keynote later than evening. It worked then.

    • vkd108 - 10 years ago

      Another example of dumbed-down grammar at work – the use of the word “than” in the above.

  3. border1ine - 10 years ago

    This guy has no idea what he is talking about, first of all, I had video problems on my appleTV, the video dropped out dozens of times. Second, the JSON requests are separate from the page. The page itself is 100% cacheable. The JavaScript code on the page will make requests for additional data, and that may or may not be cacheable, but that has nothing to do with the video. But, when I tried to watch the video in Safari, the interactive elements continued while the video itself would not load or it stuttered.
    I have been developing web applications since the 90s, and this guy is full of shit.

    • Aria Moor (@MoorAria) - 10 years ago

      I have to totally agree. This guy has no clue. It just doesn’t work that way.

      For one, when the idiot who mix the audio/video does not pay attention to the outgoing feed and mix’s mandarin translation with live audio feed, that has nothing to do with CDN or the page loading or javascript or anything Dan Rayburn might think he knows about.

      • William - 10 years ago

        Yep. Any theories what did happen with the translation? I don’t quite understand how different streams could get mixed up. (I wonder if it disappeared again because they realised there was a problem and eventually decided to revert to English only *everywhere*…)

        I do think it would be a helpful to superimpose a clock over the test card given the risk of HTTP streams arriving out of sequence.

    • vkd108 - 10 years ago

      Yes, the person(s) responsible for the live feed were completely outside of their (limited) limits. They should most definitely have their testicles scrubbed with a wire brush whilst being forced to sing “It’s my party and I’ll cry if I want to” by whoever it was.

  4. finngodo - 10 years ago

    Ran into the same issues mentioned below on Apple TV- but about 30 minutes in when they got rid of the translator audio feed I was able to airplay the rest of it without any problems.

    More than the inability for the stream to resolve, taking 30 minutes to realize that something is wonky with the audio feed is crazy.

  5. timothyrlewis - 10 years ago

    I found the countdown timer on the Apple home page to be one hour out so the stream started more than half way through the presentation. Not what I expected from Apple, they are usually quite careful about media presentation.

  6. herb02135go - 10 years ago

    I missed the announcement for iBacon. Darn.

  7. PMZanetti - 10 years ago

    It was like every other live stream that has record numbers of people tuning in….poor. Until people gave up halfway through, and it became watchable.

    • OneOkami (@OneOkami) - 10 years ago

      I was able watch the live stream of Google I/O just fine.

      • originaldub - 10 years ago

        Who cares.

      • Jack Gnasty - 10 years ago

        You and the 5 other interested people. They could have held it on a Goto meeting…

  8. If you kept trying enough times you would get to watch for about 5 minutes at a time before it booted you.

    None of their live events have ever streamed well. I’ve come to expect issues when watching any Apple live event over web/Apple TV.

    • vkd108 - 10 years ago

      The last event streamed OK, what’s your problem?

      • The last event did not stream well.

      • It dropped me out about every 3 minutes and then you had to retry about 10 times before it would pick up again.

        More annoying was the AppleTV would simply lockup at a black screen after getting an error at times which required me to unplug the device and plug it back in again to get it back up and running (happened on 3 separate occasions). I tried from the web but could never get it to load that way (I’m on a 50/10 mbps connection so it’s not like I was bandwidth starved).

      • grr… every 3 to 5 minutes…

  9. Sumeet Dodoz Bastakoti - 10 years ago

    it was pathetic….

    • originaldub - 10 years ago

      lmao, whats pathetic is all the whinging that everyone is doing.

      • vkd108 - 10 years ago

        Not necessarily. It is valid. I am in Brazil so presumed that may have had something to do with it; now I see from comments here that it was absolutely everyone hearing the oriental translator etc., it takes on larger proportions of diabolical-ness.

      • originaldub - 10 years ago

        @vkd108 I get it that someone messed up yesterday and it took a bit of time to get it sorted but it did get fixed and was totally fine after that. It’s not the end of the world and in the big picture of things there are a lot of other more important things that are worth getting truly upset about. jmo. Diabolical-ness?

      • I’m curious to know what “whinging” is.

      • originaldub - 10 years ago

        whinge (hwnj, wnj)
        intr.v. whinged, whing·ing, whing·es Chiefly British
        To complain or protest, especially in an annoying or persistent manner.

  10. And I thought the Chinese overdub was payback for all the Chinese video leaks we saw just a few days prior to the keynote.

  11. Roger J. Caldwell - 10 years ago

    For me it was 7 minutes in and the show hadn’t started. When it finally did it jumped 7.35 minutes into the “live” event complete with the stupid wench translator. Tried restarting the AppleTV, tried viewing on my iPad and both would jump 7.35 mins in and refuse to back up any further. Frustration set in and I read Mac magazine websites for the updates. Finally at around 1800 CST I was able to view the event in it’s entirety. I really love the Apple products, but it would be much nicer if they could make their streaming events perform at the same quality.

    • originaldub - 10 years ago

      Your post would have been much nicer if you didn’t write; “complete with the stupid wench translator ” Not sure what purpose it serves other than making you look ignorant.

  12. Anabel Vale - 10 years ago

    It ruined the event for me. A pity.

    BUT most importantly and shocking was the lack of technical details about the Watch, despite early 2015 launch…

  13. b9bot - 10 years ago

    Whatever the reason the stream failed horribly. Neither the Apple TV or the web on the computer worked at all. I hope they do not do it that way for future keynotes. I had to watch it later when it was posted from a recording. Everyone knows java sucks so I think he’s right about what caused the issues. In any case I hope they forget about blogging and streaming at the same time because it ruined the live stream completely.

    • iFernando - 10 years ago

      Whoever was in charge of Apple’s live stream probably doesn’t have a job anymore. Plus, Apple will spend whatever it takes to make sure that this doesn’t happen again.

      • vkd108 - 10 years ago

        Hmph. So you think, fanboi.

  14. mattsellars - 10 years ago

    my Apple TV kept crashing… it was a pretty bad start to the event luckily the live blogs kept up just fine.

  15. I wish Steve was around to fire the people that bungled this.

    • originaldub - 10 years ago

      What a lame wish.

      • Jack Gnasty - 10 years ago

        ..says the lame troll.

      • originaldub - 10 years ago

        Ok Jack… Tell me how that isn’t a lame wish?

    • herb02135go - 10 years ago

      I’m sure those employees won’t be affected by thenanti-poaching

  16. Mark Granger - 10 years ago

    How did the stream manage to crash the Apple TV? It sounds like many people experienced this problem. I had to physically unplug the power from the Apple TV to restore it to operation. This showed that Apple has a bigger problem than just a poor stream. They have buggy software running on their Apple TV.

    • Evan Huff (@evanh86) - 10 years ago

      No Doubt! I’ve noticed that when using air play from my computer iTunes to the Apple TV when playing movies many times.

  17. Richard Israel - 10 years ago

    Was it about a witch or a watch? I could,int understand the witch and I don,t understand much about the watch, Which will Apple Pay For? the Witch?or the watch, I tried to watch the Witch which was,int the Watch…never did see the Witch…..

  18. Evan Huff (@evanh86) - 10 years ago

    Their blog posts were shotty at best too… so that didn’t even really help.

  19. vkd108 - 10 years ago

    Over and above the mess, crazed conspiracy theorists noted the following possible “illuminati” (or whatever) tie-ins, linking Apple to the conspiracy (or whatever).

    1. Mega Evil Corporation – promoting their game.

    2. Monarch butterfly image, highlighted for it being female, which increases the conspiracy fervour as it is generally female persons that are subjected to the Monarch Mind Control programs, according to conspiracy theorists at least.

  20. Tom Who (@TommieWho) - 10 years ago

    There were probably many more watching the live stream than were actually at the event, so, it made Apple look bad and didn’t inspire confidence in their technical ability. Perhaps this is why Steve Jobs did not want to broadcast keynotes live. He was likely rolling in his grave yesterday. Apple set up a great event and the live feed made it seem like Amateur Hour.

  21. kevicosuave - 10 years ago

    I’ve done a lot of large live webcasts since the mid-90s. Many of them for large corporate clients. I’ve live webcast Apple keynotes as well. I was in no way involved in yesterday’s webcast (so don’t hunt me down), but here are some thoughts…

    1) Chinese woman from hell: This sounds like an easy thing to fix, but depending on the set up, this may have been very difficult to resolve. Preventing this on the other hand should’ve been easy. The problem here (which will be a common theme) is that there probably wasn’t production isolation. I’ve seen this at Apple events before. There’s an area where sound/video is provided to a bunch of crews from various media organizations to tap into. A ton of people show up just before the show and start plugging stuff in. Any number of people could’ve plugged the output, into the feed resulting in an insertion of the Chinese woman’s voice. If the live webcast source wasn’t isolated from the feed pool, then it would’ve meant that the technical director (TD) would’ve had to have either unplugged everybody, or track down who the offender was. The former would piss off a lot of media companies and the latter would’ve taken time (30 minutes seems likely in this scenario).

    2) Drop outs / restarts: I agree with Dan Rayburn, this wasn’t a CDN issue. In other words, from the video stream itself, it wasn’t that too many people were connected. There was likely an issue with connectivity from the event to the server. While connections are usually tested well in advance, often they aren’t isolated. I’ve had numerous webcasts become “challenging” due to a requirement for a dedicated line turn out to be a shared line that’s not discovered until after the event begins. Suddenly the reliable 20mbps upstream becomes an unreliable Xkbps. The thought that this was at least one of the things that went wrong comes from the fact that the streaming deaths were universally reported. MTR reporting that I did from various locations all showed clear traffic.

    3) Server/client code issues: A lot of this is covered by Dan Rayburn, but with the Apple TV it seems as if the stream would die, and then the server was swamped with too many clients trying to reconnect at once. Although, it may have also been an issue with connection issues during periods of rebooting. Again with an event like this, I’m not sure why outgoing feeds weren’t isolated, not just in the dedicated connection, but with dedicated streams to dedicated servers with multiple channels feeding the CDN.

    Whatever went wrong, it went spectacularly wrong. Apple needs to really go through this and figure this out for the next event. Too many press people from around the world were trying to use the live feed as a means of reporting.

  22. David (@davidr_ch) - 10 years ago

    For me it was clearly a problem of technical coordination.

    1) Errors on cache feed was maybe the case of delays, but a good coordination with different teams involved and right choice of method of transmission with good backups solutions can avoid these lacks to support millions of viewers.

    2) Lacks on cache are not only the problem, when some time viewvers receive the demo video of encoders, is not a technical issue, but more error of operators that receive wrong instructions after someone detect problems on stream.

    3) Another thing about the audio mixed feed with Chinese translation. This problem is a common error when you work with several languages and you use stereo channels and surround channels to send translation instead split channels in encoders correctly.

    I never imagine Apple suppliers can do errors like that, during a live event.
    Work of amateur. Think to rehearsal next time…

Author

Avatar for Zac Hall Zac Hall

Zac covers Apple news, hosts the 9to5Mac Happy Hour podcast, and created SpaceExplored.com.