Page 4 of 7

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Tue Aug 26, 2014 1:07 pm
by ramonc
For what it is worth department workaround- I just created a new calendar in google calendar - SLC-Manual. Downloaded the .ics file from the lds.org link and then imported into my SLC-Manual. Come Saturday night I will delete my google calendar SLC-Manual, Create it again, download from lds.org again and import it again. I will just keep doing this as needed until issue is resolved, thus I am somewhat current for my uses anyway.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Tue Aug 26, 2014 1:12 pm
by russellhltn
ramonc wrote:Come Saturday night I will delete my google calendar SLC-Manual, Create it again, download from lds.org again and import it again.

I wonder if it's necessary to delete the calendar. Have you tried re-importing into the existing calendar?

My primary use is to get the sync onto my Android device. Unlike iOS, Android does not have a way to do it natively. But there are apps that can be installed. One possible advantage - you can sync as fits your schedule rather than wait until Google gets around to it.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Thu Aug 28, 2014 8:11 pm
by brucejay
This is a problem on Google's end. I have had this problem before, and have it now. Google is not able to communicate with the server. However, a browser can communicate with the server and download the calendar to an ics file that can be imported into the Google calendar. The questions and comments need to be directed to Google as to the sync process.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Fri Aug 29, 2014 12:16 am
by russellhltn
brucejay wrote:This is a problem on Google's end. I have had this problem before, and have it now. Google is not able to communicate with the server. However, a browser can communicate with the server and download the calendar to an ics file that can be imported into the Google calendar. The questions and comments need to be directed to Google as to the sync process.


It may be an issue of interpreting standards. Robots.txt is a file used by the site to direct web crawlers what is and is not to be indexed. (See Robots exclusion standard) The question is, should Google pay attention to Robots.txt when being directed by a specific user to retrieve an ICS file from a specific location for their own use? It's understandable that the church doesn't want Google to index everyone's calendar into their search engine, but there's no intent to block what Google does in behalf of a specific request from a user.

Web browsers ignore robots.txt as they are not "robots" but work as directed by the user.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Sun Aug 31, 2014 10:28 am
by rbraithwaite1995
So basically nothing has been resolved on this issue?

To the person who recommended to delete the "robot.txt" from the HTML. Can you be more specific about how this is done?

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Sun Aug 31, 2014 10:48 am
by aebrown
rbraithwaite1995 wrote:So basically nothing has been resolved on this issue?

To the person who recommended to delete the "robot.txt" from the HTML. Can you be more specific about how this is done?

Whoever made that suggestion is a bit confused. The file robots.txt (it has an s on that "robots", by the way) is not in the HTML. It is a separate file. And since it is one file for LDS.org, you can't just remove it from the Calendar sync service without affecting more than just sync URLs.

Search engines gather the information for their search indexes by "crawling" the web using applications known as "web crawlers" or "spiders" or "robots" to find pages of content, then they put that information in indexes so that they can search for that content. The robots.txt file allows sites to tell search engines to ignore anything they find in certain parts of their site.

The robots.txt file for LDS.org can be found at https://www.lds.org/robots.txt . If you examine the file, you'll see a line that says "Disallow: /church-calendar". That means that web crawlers are supposed to ignore everything found in lds.org/church-calendar and below. So if we were only talking about web crawling, then Google would be correct in ignoring sync URLs, since they are found below lds.org/church-calendar. But reading a sync URL has nothing to do with crawling the web, so Google's Calendar application is misinterpreting the purpose of robots.txt if they read that and assume that they can't open a file from there.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Sun Aug 31, 2014 11:32 am
by russellhltn
aebrown wrote:Google's Calendar application is misinterpreting the purpose of robots.txt if they read that and assume that they can't open a file from there.

I'm not so sure. When I look up the purpose of Robots.txt, it seems to be aimed at automated processes, not just search engine web crawlers.

When I look for complaints about the robots.txt for Google calendar - I can find them, but they are from years ago. And thanks to the Wayback Machine we can see that the there was a change to the church's robots.txt done on Aug 13. So while we can argue standards, it was the church that made the change that broke things. The fact it hasn't been changed back makes me think there's a security issue involved.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Mon Sep 01, 2014 12:55 pm
by diidu
I have had LDS calendar/Google calendar sync issue from July or so. Today started working on it and finally recreated the link. At the first attempt of adding the new link to Google calendar I saw the mentioned robots.txt error for the first time. After that I did not see the error anymore, but no sync either. Copying the link to browser window displays an empty calendar in Firefox.

Another issue (should probably start a new thread for this, but I'm not doing it now) is that for few montyhs already I have not received any email notifications from lds calendar even though based on my settings and events added I should have.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Mon Sep 01, 2014 12:57 pm
by diidu
>> Copying the link to browser window displays an empty calendar in Firefox.

With this I mean the link for embedded calendar.

Re: Google Calendar Sync Issue (Robots.txt)

Posted: Mon Sep 01, 2014 1:09 pm
by russellhltn
diidu wrote:After that I did not see the error anymore, but no sync either. Copying the link to browser window displays an empty calendar in Firefox.

How long did you wait? There's a 9 minute throttle on the calendar. If you try to use it within the 9 minutes, you'll get a http error (although many things won't report that to you.) In most browsers, you'll end up with nothing. Creating a new URL won't reset the timer - it's tied to your account.

diidu wrote:Another issue (should probably start a new thread for this, but I'm not doing it now) is that for few montyhs already I have not received any email notifications from lds calendar even though based on my settings and events added I should have.

It's addressed in the LDS.org calendar not sending email updates thread.