Pages

Monday, June 14, 2010

Further Support for Migrating FTP Blogs

During the FTP Migration, we maintained a dedicated Issue Tracker to log and resolve issues related to the FTP Migration. It remained open for several weeks after FTP publishing formally shut down, and on May 19th we stopped accepting new issues. Comments on posts here on the Blogger FTP blog have slowed to just one or two per week, and in each case concerned specific issues with the commenter’s blog.

Moving forward, if you encounter a problem migrating your blog off of FTP publishing, please post to the “Something is Broken” section of the Blogger Help Forum, where the Blogger community (including Top Contributors and Blogger team members) will be able to address your problem. Comments are now closed on this blog, so that any outstanding issues will be posted and resolved in one place.

As always, thank you for using Blogger.

Sunday, May 2, 2010

Migrating off of FTP after May 1

As planned, we turned off support for FTP publishing last night. For those of you who have not yet migrated your blogs, we have updated the Migration Tool to allow you to continue to update your blog.

If you still have an FTP blog in your account, you will see an alert on your Blogger Dashboard. Follow that link to the Migration Tool, then select the blog you wish to migrate off of FTP.


After you download a backup of your blog, you will be prompted to select a new URL for your blog: either at your own domain or at blogspot.com. If you wish to have your blog hosted at a subdomain of your domain, you will need to create the CNAME before proceeding; details for doing so are here.



Once you make your selection (and if necessary, create your CNAME), you will download a ZIP file containing a full archive of your blog, with updated pointers to your new URL included in the HTML for each post.

Download and extract the files from the ZIP file, then upload the files to your FTP host. Once that upload is completed, you will be done with your migration. You can now create new posts and edit old posts from within Blogger, and visitors to your old URLs will be redirected to the new location.


If you encounter problems with the Migration Tool, please file an issue in the Issue Tracker and a Blogger team member will investigate.

Wednesday, March 17, 2010

Advanced setup: moving from www to www

A number of users have asked about preserving their current URL. We opted not to support this scenario with the Migration Tool because of the risk of breaking file URLs in the process -- but several of you have asked about the best way to do this and I promised a blog post documenting it.

Scenario: You host a blog at www.myblog.com, published via FTP. You host your blog (and only your blog) at this address, and you want to continue to use Blogger to publish to www.myblog.com using Blogger's Custom Domains feature. This walk-through will identify the steps needed to do that. Keep in mind that any files you previously hosted on www.myblog.com -- like uploaded images, PDFs, etc., whether they're www.myblog.com/resume.pdf or www.myblog.com/uploads/images/foo.jpg -- are currently hosted at www.myblog.com too. Note: When you update your domain so that www.myblog.com points to Blogger's servers, any references to those files will break unless you follow these instructions.

1. Create your "missing files" CNAME. This will be a secondary domain you will create that will act as a backup for requests that go to Blogger which result in a 404.  (More on this issue here.) To create the CNAME, pick a name for this backup domain ("files" is a popular choice) and point it at your webhost's IP address. If you have a question about setting up this CNAME, check with your domain registrar and/or webhost. (It's possible, through a tool like CPanel, that your webhost has simplified tools for this step. Check with them.)


Important: pick an image that currently works at www.myblog.com and verify that it also works at files.myblog.com before proceeding. It may take up to a few hours for DNS to propagate; be patient.

2. At Blogger.com, click on "Settings | Publishing" and click on "Switch to: Custom Domain".


Click "Advanced":

and type in your domain ("www.myblog.com").




3. Click "yes" under "Use a Missing Files Host" and type in the URL you picked in step 1.


4. Fill in the word verification, and click "save settings".

5. Now update the CNAME for 'www' at your domain registrar, so that 'www' points to 'ghs.google.com.' (The trailing period is important.)

Once you click "save settings", Blogger will immediately start serving requests it receives for www.myblog.com. There may be a delay of a few hours as your DNS changes propagate. That's it -- by following these steps you will successfully convert your FTP blog into a Blogger Custom Domain blog and keep the same URL you had before.

Monday, March 8, 2010

New process for reporting issues

We've been working to answer questions over the last few weeks, and when the migration tool launched last week, we set up a spreadsheet to take in reports of problems with the tool.

Unfortunately, that's created a bit of a mess. Users were reporting problems with the tool in comments (in some cases on posts that are weeks old), questions were being asked about things not related to the tool in the spreadsheet... and members of the Blogger team who wanted to chip in and help out didn't know where to look for the most recent reports of problems.

Starting this afternoon, we moved all reports of issues from the spreadsheet form to a dedicated issues tracker. We've been evaluating this as a way to improve user support more generally on Blogger, and now seemed a perfect time to test out this approach to see if it helps. So far it seems to be helping, so we'd like to provide a quick update on how we will be supporting this going forward:

  • Effective immediately, I've opened up comments on the FAQ page for general questions. Questions that get asked repeatedly will get bumped up to the FAQ page itself.
  • I'm closing old comment threads and directing general questions to the FAQ and specific problems to the issue tracker.
If you file an issue, you will get notified any time we update the ticket with additional info. If you search the open issues and find one that you're interested in, simply star that issue and you'll receive e-mails each time the issue is updated.

To anyone who filled out the form over the weekend, you can search the issues tracker for your URL and you will find your ticket. Feel free to update the ticket with any additional details. (Note that we may have marked the issue fixed if we determined that things were working properly; if you don't see your ticket, search all issues and not just open issues.)

Wednesday, March 3, 2010

FTP Migration Tool Released

Earlier this evening we released the FTP Migration Tool. We are initially releasing the tool to Blogger in Draft so that we can closely observe the tool’s performance before releasing it to all users at www.blogger.com. If you’d like to give it a try, you may go to Blogger in Draft; if you have one or more FTP blogs in your account, you’ll see this alert:




This screencast walks you through the process:





A few notes about the tool:
  • The final post is in English only; translated versions for non-English blogs will be available next week
  • Once the migration is complete, you can delete the final post from your blog’s dashboard by going to Edit Posts and selecting the post for deletion. It will remain on your FTP blog so that visitors to that blog will know about the new location.
  • Once you migrate your blog to Blogger hosting, you won't be able to return it to FTP publishing
If you encounter any problems during your use of the migration tool, we’re collecting reports of problems here and will continue to reach out directly to help resolve any outstanding issues.

Migration tool update: launching tonight

A more complete post will follow when the tool is available in production later tonight, but we wanted to let you know that we are in the process of pushing the latest Blogger build which contains the FTP Migration Tool. Before we release it, we will do a final sanity test to ensure that everything behaves as it should - and expect to push it first to Draft and a short time later to www.

I did a screencast of the tool to give you an idea of how this will work. This assumes you're moving from an FTP blog to a Custom Domain; we'll do a separate screencast (likely later tonight) showing the same process but moving to a Blogspot URL.

Friday, February 26, 2010

Migration tool postponed until first part of next week

We've been working around the clock to get the migration tool launched this week as originally promised, and ran into two bugs which we are in the process of correcting. We are not happy about delaying this by a couple days, but launching it with the knowledge that a few bugs remained is definitely not the way to go into a weekend.

We'll have the video walkthrough and announcement post on this blog as soon as it's ready.

Monday, February 22, 2010

Migration tool almost ready, still on track

When we announced the migration tool last month, we pointed to the week of February 22 for its launch. That week is upon us, and we're just about ready to launch. We're doing some final QA on the tool, and will get it launched this week. Exact launch date is subject to a few variables - but we will update the blog as soon as it's available.

Once launched, you'll see an alert in your dashboard if you have any FTP blogs to migrate, and will be able to get started right away.

Friday, February 12, 2010

Migration deadline extended to May 1

When we originally announced that we were shutting down FTP publishing on Blogger, one of the common complaints we heard was that March 26 was too soon. Many of you asked for more time to manage the transition off of FTP publishing, either to migrate your blog(s) to Blogger hosting or to another platform.

March 26 was not an arbitrary date  — it was related to an upcoming infrastructure change being made at Google that will render our current FTP code unworkable. We relayed your feedback to the infrastructure team, and they have agreed to postpone their change, which allows us to extend the deadline to May 1.

We are finishing up the FTP migration tool, and are still working to have it out the week of February 22.

Finally, we've done a sweep through your recent comments, and will be updating the FAQ later today with more answers.

Thursday, February 11, 2010

The Technical Side of FTP

by Noah Fiedel, Blogger Engineering Tech Lead

Many of you have posted asking why Google doesn't continue supporting FTP publishing indefinitely or even charge money to those who would like this support. I'd like to give you some insight into our decision making and technology.

The Protocol

FTP is one of the earliest protocols on the Internet. It was drafted in 1971 before security was a concern, as the Internet at the time connected universities and research labs. Unlike nearly all other Internet protocols, it uses two insecure and unencrypted ports simultaneously. This makes securing FTP effectively impossible on both the server and network levels. FTP servers at ISPs are therefore vulnerable to attack, and your password can be 'sniffed' by anyone with access to the traffic to or within your ISP. sFTP, while more secure than FTP, still requires us to store your user credentials — which itself is undesirable from a security perspective.

Compare this to the HTTP protocol, drafted 20 years later in 1991: FTP doesn't have a mechanism to discover whether an FTP server is up, down, slow, or temporarily unavailable. HTTP supports all of these and more, and is now the basis for nearly all activity on the Internet.

Due to FTP's weaknesses, many ISPs restrict access to their FTP servers. They do this by limiting your FTP account to a list of approved Internet addresses (via an IP whitelist), which makes your account less likely to be hijacked. In the next section you will see why this also makes it difficult for Blogger to reliably provide FTP publishing.

Google Datacenters

Google runs many datacenters, and Blogger runs in several of them. Each datacenter has a different Internet address (a.k.a. "IP address") when it connects to your FTP server, so if your hosting service requires an IP whitelist, you would have to list all of the IP addresses associated with each of our active datacenters. We really don't like having a "primary" datacenter for anything, and instead prefer to let our traffic flow to the most efficient and lowest latency datacenter for our users. This makes the IP whitelist problem even worse, as some ISPs only allow a single (or very few) IPs to be whitelisted. Your ISP would need to whitelist all of Blogger's datacenters. Since they change regularly, your ISP's whitelist would have to be updated as well. This leads to a significant amount of user frustration, and regularly results in blogs failing to publish successfully. Diagnosing these issues has taken up a large part of our engineering and support team's time.

FTP Web Hosting Providers

Late last year during scheduled maintenance on a datacenter, we moved FTP publishing to another datacenter and updated our publicly posted IPs for your ISP's whitelists. Even after doing this, there were considerable complaints by users unable to publish via FTP to thousands of ISPs. Many of these ISPs maintain their own IP whitelists, often in an undocumented way. Troubleshooting this is extremely difficult and time consuming for us (and for you), as it's rarely clear where the underlying issue is. Our engineering, product and support teams often ended up directly contacting ISPs, waiting on hold, frequently without resolution. All to support a single user's report of "can't publish via FTP". In many cases the user or ISP simply entered the IP whitelist incorrectly. In other cases the hosting service's FTP server was unreachable. On more than one occasion, the ISP had set up "staging" and "production" environments without telling their users what was happening, so while Blogger was successfully publishing (to the staging server), the posts were not visible on the web and the user had no idea why their posts weren't showing up. In a great deal of cases, FTP publishing works but is extremely slow due to shared hosting plans having slow or limited network or disk per user. We have seen cases of full FTP republishes taking over a month, entirely due to the FTP server being slow.

What about sFTP?

If sFTP addresses some of the concerns with FTP, why are we shutting it down too? Fewer than 15% of users have adopted sFTP as their publishing mechanism, and many of the same challenges apply to both sFTP and FTP. Even with sFTP, a republish of a blog can take longer than a month to complete. Not all ISPs support sFTP and of those that do, many lock down FTP and sFTP with the same IP whitelist.

Engineering Effort

Blogger's current FTP support was 100% re-written for stability and maintainability in 2008. We added redundant queues on our side to make sure we never missed a file. We spent a significant amount of engineering time improving FTP support in the last two years, not including support and troubleshooting user issues. Even after this effort, approximately 10% of all FTP publishes fail.

As the original blog post mentioned, Google infrastructure is changing and would require us to again rewrite a major portion of our FTP support. Even after that rewrite, things would be no better than they are today in terms of stability, and we would be running just to stay in place.

Decision

I hope this post helps you understand the issues we face, and why it is not simply a question of money or a small bit of time. We want to deliver a best-in-class product experience for all users. Supporting a protocol with known security vulnerabilities and dependencies on downstream ISPs was preventing us from delivering the stable, reliable, and functional product we want and our users demand. It was also preventing us from doing more for the 99.5% of users who host their blogs with us, either on their own domain or on blogspot.com. While we deeply regret the impact this has on some of our users, many of whom have relied on Blogger for years, we remain confident that this was the right decision.

Wednesday, February 3, 2010

Who's affected?

There has been some confusion in the comments about who is impacted by the announcement that we're deprecating FTP on Blogger. The easiest way to determine if you're affected is to go to Blogger, select "settings" for your blog and click on "publishing". If you're on FTP, you'll see something like this:


A bit more information:
  • If your blog URL ends in "blogspot.com", you are not affected
  • If you don't see "You're publishing via FTP" (or "You're publishing via S/FTP") when you go to "Settings | Publishing", you are not affected

Missing Files Host: what it does, why it helps

For people contemplating a switch to custom domains, one good question that's shown up repeatedly in the comments concerns files other than blog posts. These could be images, PDFs, Word docs, etc. - and if you're moving to a custom domain, you will want to think about what happens to those files.

Before we get to the specifics of what's happening, it's useful to take a step back and understand the mechanics of DNS and how it relates to your blog. Let's assume for a moment your blog is published to http://www.yourdomain.com/. There are several parts to that URL that are important for this discussion:

  • yourdomain.com: The domain you registered, from your registrar.
  • www: the CNAME (often referred to as the subdomain), commonly configured by default by your registrar, to point to a nameserver.
  • Nameserver: the server that stores the right IP address to direct users to for the CNAME
  • IP Address: the numerical address (something like 123.45.6.7) that identifies the server which hosts the content

When someone types "www.yourdomain.com" into their browser, the CNAME is passed to the nameserver, which translates the CNAME into an IP address, which then receives the request (for the homepage, for instance) and sends it back.
Important note: For this illustration, we're using the example of someone who's moving from www.yourdomain.com (hosted by someone else) to www.yourdomain.com (hosted by Blogger). If you are considering setting your blog up on a subdomain (i.e., blog.yourdomain.com), your setup will be slightly different. The purpose of this post is to explain Blogger's Missing Files Host; later posts (and the migration tool) will provide more guidance about addressing specific situations.
When you move a CNAME (in this example, let's assume that you were hosting www.yourdomain.com with "Joe's webhost", and have opted to move www.yourdomain.com to Blogger), you're simply instructing the Internet to direct requests for "www" to Blogger's IP address, not Joe's Webhost.

The important thing to recognize at this point is that Joe's Webhost still has your old content, but there is no URL to request the old content. Blogger's Missing Files Host is designed to address this, by watching for requests to us that 404 (meaning we can't find them) and rewriting them to look for them in another location.

Using the above example, let's spell this out:

  • Old setup: www.yourdomain.com --> 123.45.6.7 (maintained by Joe's Webhost)
  • New setup: CNAME "www" to point to Blogger, which results in requests for www.yourdomain.com --> Blogger's IP address
  • Create a CNAME (we'll use "files") to point to Joe's Webhost (Joe's Webhost can help you with this, our help file discussing CNAMEs is here)
  • In Blogger's Custom Domain options, enable the Missing Files Host and input "http://files.yourdomain.com"
Result? You have a PDF stored on your domain today (managed by Joe) at http://www.yourdomain.com/uploads/resume.pdf. Once you point "www" to Blogger, requests for that URL will fail - we don't have that content on our servers. By enabling the Missing Files Host, requests will first go to us, and when we can't find the content in question, we'll automatically redirect the request to the backup URL, in this case: http://files.yourdomain.com/uploads/resume.pdf. This will be invisible to the end user, and will ensure that all of your old content will surface as intended.

Tuesday, February 2, 2010

FAQ: SFTP counts as FTP

Addressing a common question I’ve heard: SFTP support is also being discontinued along with FTP.

We’re playing a bit fast-and-loose with the terminology of these actually quite-different protocols, so whenever you read us saying something about “FTP” just add in “and SFTP” to it and you’re set.

For answers to this and other questions, take a look at our FTP (and SFTP) FAQ.

FAQ page published

Just added an FAQ page to address some of the most common questions.

For blogs that are no longer updated

If you no longer update your blog, but it is still configured to publish via FTP, then you received an e-mail from us today announcing the FTP shut-down. If you don't intend to update your blog, then you don't have to do anything. You won't have the ability to update the blog from within Blogger, but if you don't intend to publish new content to the blog, the HTML files will remain intact on your server and your existing posts will continue to function just fine.

If, on the other hand, you intend to update the blog down the road, you will be able to use our migration tool to convert your FTP blog to a Blogger-hosted blog (either at your own domain or at Blogspot), and you can then update the blog at a later date.

Some background on the process

Long-time Blogger engineer Pete Hopkins wrote a great blog post last week on his personal blog that captures the thought process behind our approach to shutting down support for FTP. From his post:
Though we’ve tried to put together a migration process that will work smoothly for everyone, I’m sure it won’t be perfect; there are too many moving parts in FTP publishing to guarantee that everyone will have a great experience. Nevertheless, I believe that our overall plan is sound, so I’d like to tell you about what we came up with, as well as some of the alternatives that we considered (and that might work better for you if you want to try them out).
That last part is key: he outlines a number of more "advanced" approaches that may be more appropriate for users, given certain situations. We'll be talking more about those in the next couple weeks here, and will answer questions as you raise them.

If you're interested in understanding more of the thinking behind what we're doing and why we're doing it, give Pete's post a read.

Wednesday, January 27, 2010

Migration tool overview

Last week's announcement mentioned a migration tool we will be releasing in February, and a number of people have asked what it will do. In this post, I'll give a summary of the functionality, and then answer questions in the comments.

The goals of this tool are straightforward. It will:
  • ensure that visitors to your FTP blog are redirected to your blog's new home
  • avoid having pages from both sites end up indexed in search engines
  • preserve PageRank from your existing site to your new URL
  • update your Blogger settings so that future posts publish automatically
  • identify any necessary changes you need to helps you through each one
The tool provides a wizard-style interface which walks you through each of the four steps of the process:
  1. Backup your blog. Blogger supports full blog backup/export today, and before we kick things off we want you to have a full copy of your blog.
  2. Next, the migration tool asks you where you want to publish the blog once it's hosted by Blogger. This could be at a subdomain of your current domain (if you currently publish to www.yourdomain.com/blog/, you might want the blog to reside at blog.yourdomain.com), or if you don't own the domain you're publishing to, it might just be a Blogspot URL. If you pick a subdomain, we'll instruct you how to update your DNS settings so that the CNAME properly points to Google's servers.
  3. Once your blog is available at the new URL, we will write a final post to the FTP blog instructing users to the new URL, and we will then update your existing FTP pages by adding two pieces of information to each page on the blog: one is called a rel="canonical" link, which tells Google that the "canonical" location of the page is no longer the FTP URL but the new Blogger-hosted URL. The other piece of info is a refresh line, which tells the browser to redirect the visitor from the page they're on to the canonical URL after a short pause. Now that rel="canonical" supports moving content across domains, this is the path that Matt Cutts recommended we use to ensure that we could simply redirect all traffic from the FTP URL to the new URL. (More on rel="canonical" here, and more on refresh here.)
  4. Finally, we validate that the FTP URLs resolve to the new URLs — we just pull a couple of randomly selected blog posts and verify that they properly resolve to the new URL.
Most users will be able to use this tool to minimize the disruption of migrating off of FTP. Once done, you will continue to publish from Blogger — except that instead of watching as multiple files get uploaded to your webserver, your post will be immediately visible on the web and all archives pages are rebuilt automatically.

Some users will want to exercise more control over the process — we're also building a checklist to help identify all of the things you'll want to make sure you cover in manually moving your site off of FTP. More on that in a later post.

Tuesday, January 26, 2010

Updates from some FTP converts

Following last week's announcement, a couple of users have shared their experience moving off of FTP to Blogger's Custom Domain feature. We thought their write-up of their experience, along with their early reactions to the switch, was worth sharing.

Kent Newsome:
I've now experimented with Blogger Custom Domains and the newer features enough to confidently report that publishing via Custom Domains is a reasonably powerful platform.  The inclusion of static pages (via the Blogger in Draft beta page) adds the much needed ability to include ancillary pages.  See the index pages I added to Errbear.Com for an example of how to implement static pages.  I also found it reasonably easy to modify the new template, as you will see.  It's early, but so far I'm pretty impressed.

Conclusion
Overall, this was a pretty easy process.  I don't know if I the additional Blogger features that weren't available with a custom template will outweigh the limitations of a canned template, but I can tell you that the process of moving to a Blogger Custom Domain was pretty easy.

Louis Gray:
As a recently converted FTP Blogger user, I know the experience I had on the FTP site was less robust and less dynamic than the Google experience I've long been used to. Switching, though a non-trivial process last fall, which also saw me migrate from www.louisgray.com to blog.louisgray.com, has been nothing but good news for me since, with greater access to all the new Blogger features, and near-instant publishing without outages that used to plague the FTP service.

Both Louis and Kent have offered to answer questions on their posts, and you're welcome to post a comment here as well. We will be sharing more details about the migration tool that's being built shortly, as several people have asked specifically what we will be doing when we release it in February. Stay tuned.

Friday, January 22, 2010

Deprecating FTP

Update, 3/6/10: The Migration Tool is now live for all users; a walk-through of the tool is here. If you have used the tool and run into a problem, please report them here. Comments on this post are closed; you can direct general questions about the migration process on this thread and a fellow FTP user or someone from the team will get back to you asap.


(Cross-posted from Blogger Buzz)


Last May, we discussed a number of challenges facing Blogger users who relied on FTP to publish their blogs. FTP remains a significant drain on our ability to improve Blogger: only .5% of active blogs are published via FTP — yet the percentage of our engineering resources devoted to supporting FTP vastly exceeds that. On top of this, critical infrastructure that our FTP support relies on at Google will soon become unavailable, which would require that we completely rewrite the code that handles our FTP processing.

Three years ago we launched Custom Domains to give users the simplicity of Blogger, the scalability of Google hosting, and the flexibility of hosting your blog at your own URL. Last year's post discussed the advantages of custom domains over FTP and addressed a number of reasons users have continued to use FTP publishing. (If you're interested in reading more about Custom Domains, our Help Center has a good overview of how to use them on your blog.) In evaluating the investment needed to continue supporting FTP, we have decided that we could not justify diverting further engineering resources away from building new features for all users.

For that reason, we are announcing today that we will no longer support FTP publishing in Blogger after March 26, 2010 May 1, 2010 (update: deadline extended). We realize that this will not necessarily be welcome news for some users, and we are committed to making the transition as seamless as possible. To that end:

  • We are building a migration tool that will walk users through a migration from their current URL to a Blogger-managed URL (either a Custom Domain or a Blogspot URL)that will be available to all users the week of February 22. This tool will handle redirecting traffic from the old URL to the new URL, and will handle the vast majority of situations.
  • We will be providing a dedicated blog and help documentation to provide as much information as possible to help guide users through the migration off of FTP.
  • Blogger team members will also be available to answer questions on the forum, comments on the blog, and in a few scheduled conference calls once the tool is released.
We have a number of big releases planned in 2010. While we recognize that this decision will frustrate some users, we look forward to showing you the many great things on the way. Thanks for using Blogger.