Apples Notarizing Hell
At WWDC 2018, Apple introduced a new "security" feature to macOS called Notarizing.
This announcement did concern and puzzle me a lot. Didn't we already have the super safe and amazing "codesigning", which was stuffed down our throats with Mac OS X 10.7? This codesigning was supposed to prevent dangerous code of unknown origin from being executed on your Mac.
But the introduction of codesigning by Apple was a total fiasco, riddled with bugs, failing developer tools, missing integrations, no documentation at all, and many other problems, and it took Apple four full years to get it to work more often than not.
And yes, that process was called codesigning hell, and for good reasons.
And now Notarizing.
It turns out that the hyped codesigning did NOT prevent bad things from happening, and so now Apple forces developers to upload their applications to Apple, so that they can do inexplicable things to the code, and when they think all might be fine, they give you a special stamp of approval.
If you launch an non-notarized application in macOS 10.15, you get this dangerously worded scare dialog:
Honestly, I take great offense on the wording of this. Using my application name and the word "malware" in one sentence is suggestive and extremely offensive by Apple.
In my eyes, the text of that dialog window should read: "macOS doesn't want to launch this application because it is not notarized. Please contact Apple so they can fix all the bugs in Xcode and the notarizing process."
Why? Keep reading!
The Gates of Hell, Part 1
Apple claims you can use their main developer tool Xcode to notarize your macOS application with a few simple clicks.
Truth is, that doesn't seem to be the case. For our NeoFinder project, for example, Xcode doesn't even show the options to notarize the build and archived product. And yes, we have filed a bug report about this 16 months ago, and Apple said that somehow Xcode couldn't really see that NeoFinder was actually really an application, so it probably possibly didn't really work. And that was all. No help from Apple at all beyond that point.
What a joke. And yes, that was not helpful, Apple. So far, they haven't been able to figure that out, and there is no documentation whatsoever why this may fail.
Instead of using Xcode, you can also use a series of partially documented and super complicated command line tools with loads of complex parameters to get the job done, but frankly, even though I use Terminal on Unix systems for 30 years now, that was too much for me.
Fortunately, the amazing people that bring you Script Debugger developed a mighty user interface to these whacky command line tools, called "SD Notary"
With that brilliant tool, I was able to at least start the notarizing process.
And how easy it is, thanks to Apple. ;-)
1. First, you login to https://appleid.apple.com and create an App-Specific Password. You can find that in the Generate Password segment of your account information.
2. Open Apples Keychain Access.app in the Utilities folder of your Applications folder. Select the "login" keychain in the left list, and use "New password item" in the File menu. enter the name of your app-specific password as the Keychain Item Name, enter your Apple ID for the Account Name, and enter the actual app-specific password in the Password field.
That was totally easy, right? Just a few simple clicks.
Yes, that was a joke. This process is not easy nor transparent at all.
Fortunately, the wonderful people of LateNightSoftware wrote a really good tutorial about that, something that Apple should have done in the first place.
The Gates of Hell, Part 2
Anyhow, I got SD Notary set up, and pressed "Submit App..."
But wait, what is that? An error message?
12:32:26.473: Error 105553142767344 signing '/Users/*/Library/Developer/Xcode/DerivedData/NeoFinder-dodvyesmmrunrbftbfffzezljcjz/Build/Products/Debug64/NeoFinder - Working/NeoFinder.app/Contents/Frameworks/ZXingObjC.framework/Versions/A': Developer ID Application: NORBERT DOERNER (XXXYYYZZZ): ambiguous (matches "Developer ID Application: NORBERT DOERNER (XXXYYYZZZ)" and "Developer ID Application: NORBERT DOERNER (XXXYYYZZZ)" in /Users/*/Library/Keychains/login.keychain-db)
What the heck does that even mean? Is that one of the carefully worded and helpful error messages that we developers are supposed to produce when something goes wrong? No.
Well, it turns out we see TWO Apple bugs here at once.
1. /usr/bin/codesign is unable to properly run if multiple valid "Developer ID Application" certificates from Apple are present in the local keychain.
2. Who put these certificates there? Right, it was Xcode, and no, that bug hasn't been fixed yet.
Thanks to the ever helpful Stackoverflow, I was actually able to figure that one out. The arcane error message of Apples codesign tool was NOT helpful at all.
Both bugs are actually well known to Apple, have been reported multiple times already, and you have guessed it, not being fixed. Another day wasted.
The Gates of Hell, Part 3
Now that the codesigning bugs were fixed, SD Notary was able to codesign and upload the application to Apples Linux notarisation servers.
But another error appeared quickly:
13:03:42.855: Result for /usr/bin/xcrun altool --notarize-app -f /Users/ndoerner2/Library/Developer/Xcode/DerivedData/NeoFinder-dodvyesmmrunrbftbfffzezljcjz/Build/Products/Debug64/NeoFinder - Working/NeoFinder.zip --primary-bundle-id de.wfs-apps.neofinder -u xxxxx -p @keychain:NeoFinder Notarize --output-format xml
Termination status: 24
StandardError: (null)
13:03:42.935: Error 105553142691904 in xcrun altool --notarize-app.
What was that now? Our developer account with Apple was active and working, so this seemed strange. But alas, after logging into itunesconnect, there was indeed the need to agree to some weird changes made to the agreements by Apple.
And as always with Apples web services, that option was well hidden in some deep levels of the web site, and there were actually TWO texts to be used as button links right next to each other, one for setting up account contacts, and the other one to agree to some agreement changes. It was impossible to see these were two separate buttons, so it took us some time to see what they actually wanted from us.
It seemed that these changes in the contract were necessary once the first notarizing request was received by Apple. Apples developer forums are full of people who have the same problem, but were unable to solve them.
With that out of the way, we were hopeful to get the notarizing done soon.
The Gates of Hell, Part 3
But soon is a very flexible term. In the first attempt, it took Apple more than one hour to do the inexplicable things to our code after is was uploaded, and then finally approve and notarize it. Prepare for it to take a lot longer, so this is not a quick thing you can do regularly. And prepare to keep the machine doing that running, you don't want to start from scratch when something fails.
But at least we now have a notarized NeoFinder 7.5b1 for testing in macOS 10.15.
The Gates of Hell, Part 4
Unfortunately, that is not all. The new notarizing process completely breaks plugins and bundles, and it does so by design and on purpose.
It is simply no longer possible to load Plugins or Bundles into any application unnotarized.
With Applications, a user can still choose to open it anyway (using the Open command in the context menu), but with Bundles or Plugins, that is simply not possible anymore.
That is a massive problem for all powerful applications that rely on being extensible. NeoFinder already suffers from it for its mighty AutoTags Engines.
[NSBundle load]; silently fails in macOS 10.15 if either the host application or the bundle are not properly notarized and some additional undocumented and arcane conditions are not met.
And guess what? YES! Xcode cannot even notarize our bundles, as we see the same bug we had encountered in "The Gates of Hell, Part 1" already. And yes, there is no documentation and no help from Apple about this problem either.
If you use older software with 3rd party plugins, you have a massive problem in macOS 10.15.
The Gates of Hell, Part 5
Apple is very creative to create even more disasters. Today, I was greeted by Xcode with this beautiful and helpful (no, that was a joke, it is not!) error message:
resource fork, Finder information, or similar detritus not allowed
And of course, the arcane and fragile "codesign" process has again failed miserable. This new error message again has not been documented at all by Apple, and I have no idea what causes this. Of course, it may be related to any one of the 800 files in the NeoFinder application bundle, but the message does not say which one of them.
Also, this error does not say which of the many mentioned situations actually causes this new problem for Apples developer tools.
This is the kind of high-quality software crap from Apple we have to work with lately. Some research on stackoverflow gave us a hint, and it seems this particular case was a graphic file that contained an xattr, extended attribute. We had to manually search for the offensive file in the command line, delete this xattr in the command line, and clear the build folder, or Xcode wouldn't actually pick up the changed file and keep generating this weird error.
Of course, as the files in question were copied to the "build folder" by Apples Xcode itself, why on earth did they even copy the problematic metadata in the first place, when they cannot handle them in "codesign"? Isn't anyone at Apple actually taking care of such things, and paying attention?
Conclusion
Apple has again done a horrible job with the new notarization process. And let's not talk about the possible future, in which Apple flatly denies notarization for apps that use certain APIs, or NOT use things Apple suddenly deem important.
While Apple right now claims this is NOT a review process, I am very certain they will use it as such one day.
I am not sure they fully understand what damage they currently do to the macOS ecosystem for forcing this immature technology on us developers. The developer tools are clearly incredibly buggy, not documented properly, and very badly designed, and the whole thing is again a nightmare.
I have posted all the technical details here, so maybe fellow developers find clues to fix their own Apple notarizing hell problems.
Apples developer forums are full of developer postings with loads and loads of notarizing problems.
There is ONE single Apple employee helping out to get the worst problems solved, the venerable Quinn “The Eskimo!” From what I see, he is doing that on his own time, and nobody else at Apple gives a **** about this.
At the end of the day, notarizing will not solve the malware problem at all, as Apple is of course technically unable to find and prevent ALL possible dangerous codes.
It is another failed safety promise by Apple that gives a dangerous appearance of safety, which is not the same as real safety.
And the really bad implementation of notarizing damages the macOS ecosystem massively, undermining the trust of the users in indie developers (consider the offensive wording in the scare dialog), and massively undermining the trust of developers in Apple itself.
Apple urgently needs to take action now.
They must finally begin to fix all these bugs, add a LOT of really good documentation to all the new file and folder cages they have added, and stop releasing yearly major macOS updates. They are clearly not able to handle the software quality issues present in that.
And for you: Don't yet update to macOS 10.15. That macOS version is clearly not ready to be used daily. Wait until Apple has released at least macOS 10.15.5 next July, and hopefully some of these issues are fixed by them. Like the horribly embarrassing bug in Apples Mac App Store, where you can still purchase, download, and installed old 32-bit applications. You just cannot run them. Or the massive networking problems when trying to connect to SMB servers or other Macs via File Sharing. That is the kind of low quality software we currently see from Apple.
NeoFinder 7.5 for macOS 10.15
And yes, of course, NeoFinder 7.5 is indeed notarized, and the AutoTags plugins, too. But only with massive help by the invaluable "SD Notary" tool, written by a small 3rd party development team.
How the largest company on the planet is unable to come up with acceptable developer tools for this ill-conceived process really eludes me. Maybe they are too busy counting their money. Shame on you, Apple, for this fiasco. When will you start fixing this?
Further macOS 10.15 reading:
https://furbo.org/2019/09/04/icloud-clusterfuck/
https://tyler.io/broken/
https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/