Late on Friday afternoon, my day-to-day phone, a Samsung Galaxy S II which runs CyanogenMod 7.1, stopped booting – it got stuck at the “rotating arrow” blue Android for ever. I use my phone rather heavily, so this was a significant inconvenience – doubled by the fact that on Saturday morning, I flew to the Bay Area for a week at the Mozilla office. “What’s the confirmation code for collecting your train tickets, sir?” “Where are you staying in California, sir?” Er…

The last action I took before this happened is that I made a full system backup via Recovery mode (I use the excellent ClockWorkMod Recovery). I’m not sure if that’s what caused the problem, but it was instrumental in the solution. I could still boot into Recovery, and fortunately that also allows console access via adb shell.

Things which didn’t solve it or give any clues:

• Looking to see if I’d filled a partition (I hadn’t)
• Making a second backup, watching carefully for errors
• Restoring the backup I’d made
• “Fix permissions” in the Recovery options
• Wiping the cache, dalvik cache and other non-destructive wipes
• Reinstalling the same CyanogenMod build again

After some trepidation, I tried doing a full factory reset from Recovery. This got the phone booting again, but without any of my data :-(. Fortunately, ClockWorkMod has a “partial backup restore” function. I restored the data partition… and the phone stopped booting again.

So this is progress. I now had some idea where the problem lay. I finally found it using a laborious 5-minutes-per-cycle manual bisection technique. Delete half the stuff on the data partition, reboot, if it still fails, delete more, reboot… once you get it to boot, restore the data partition, narrow it down further. Once you find a top-level directory, repeat the process inside it.

The result: the existence of a single 0-byte file stopped my phone from booting entirely. The file was:

data/system/profiles.xml

I have no idea what that file does (there are only a few references online), how a 0-byte version of it got created, or why having it existent but empty breaks things but if it’s not present everything works fine. I have no idea if anyone else will ever see this problem, or if they will ever find this blog post. But still, here’s my Wisdom of the Ancients.